No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Global Concept-Based Interpretability for Graph Neural Networks via Neuron Analysis
Abstract
Graph neural networks (GNNs) are highly effective on a variety of graph-related tasks; however, they lack interpretability and transparency. Current explainability approaches are typically local and treat GNNs as black-boxes. They do not look inside the model, inhibiting human trust in the model and explanations. Motivated by the ability of neurons to detect high-level semantic concepts in vision models, we perform a novel analysis on the behaviour of individual GNN neurons to answer questions about GNN interpretability. We propose a novel approach for producing global explanations for GNNs using neuron-level concepts to enable practitioners to have a high-level view of the model. Specifically, (i) to the best of our knowledge, this is the first work which shows that GNN neurons act as concept detectors and have strong alignment with concepts formulated as logical compositions of node degree and neighbourhood properties; (ii) we quantitatively assess the importance of detected concepts, and identify a trade-off between training duration and neuron-level interpretability; (iii) we demonstrate that our global explainability approach has advantages over the current state-of-the-art -- we can disentangle the explanation into individual interpretable concepts backed by logical descriptions, which reduces potential for bias and improves user-friendliness.
... However, most of these methods are limited to providing local explanations tailored to specific input instances or rely on interpretations based on input feature attributions (Pope et al., 2019;Ying et al., 2019;Vu & Thai, 2020;Lucic et al., 2022;Tan et al., 2022). Another line of research focuses on global explanations that describe the overall behavior of models (Xuanyuan et al., 2023;Azzolin et al., 2023). These approaches offer more human-readable and precise explanations by leveraging logical formulas and interpretable concepts. ...
... The task is to classify whether the ego in each ego network is a doctor or a nurse (Azzolin et al., 2023). For the baseline approaches, we employ a generation-based method, XGNN (Yuan et al., 2020), as well as two concept-based methods, GCNeuron (Xuanyuan et al., 2023) and GLGExplainer (Azzolin et al., 2023). All these methods are considered state-of-the-art. ...
... On the other hand, concept-based approaches aim to provide more human-readable and precise explanations by leveraging logical formulas and interpretable concepts. For example, GCneuron (Xuanyuan et al., 2023) identifies predefined concepts, formulated as logical combinations of node degrees and neighborhood properties, associated with specific neurons. Similarly, GLGExplainer (Azzolin et al., 2023) builds on local explanations from PGExplainer (Luo et al., 2020a), maps them to learned concepts and derives logic formulas from these concepts to serve as class explanations. ...
Graph neural networks (GNNs) operate over both input feature spaces and combinatorial graph structures, making it challenging to understand the rationale behind their predictions. As GNNs gain widespread popularity and demonstrate success across various domains, such as drug discovery, studying their interpretability has become a critical task. To address this, many explainability methods have been proposed, with recent efforts shifting from instance-specific explanations to global concept-based explainability. However, these approaches face several limitations, such as relying on predefined concepts and explaining only a limited set of patterns. To address this, we propose a novel framework, LOGICXGNN, for extracting interpretable logic rules from GNNs. LOGICXGNN is model-agnostic, efficient, and data-driven, eliminating the need for predefined concepts. More importantly, it can serve as a rule-based classifier and even outperform the original neural models. Its interpretability facilitates knowledge discovery, as demonstrated by its ability to extract detailed and accurate chemistry knowledge that is often overlooked by existing methods. Another key advantage of LOGICXGNN is its ability to generate new graph instances in a controlled and transparent manner, offering significant potential for applications such as drug design. We empirically demonstrate these merits through experiments on real-world datasets such as MUTAG and BBBP.
... The diversified landscape of GNN explainability research is visualized in Fig. 1. We summarize each of the categories below: , Grad-CAM [33]; Decomposition: Excitation-BP [33], GNN-LRP [38], CAM [33]; Perturbation: GNNExplainer [59], PGExplainer [30], SubgraphX [62], GEM [27], TAGExplainer [51], CF 2 [43], RCExplainer [6],CF-GNNexplainer [29], CLEAR [31]; Surrogate: GraphLime [18], Relex [64], PGM-Explainer [47]; Global: XGNN [60], GLG-Explainer [5], Xuanyuan et al. [54], GCFExplainer [19]. ...
... • Model-level: Model-level or global explanations [60, 19,54] are concerned about the overall behavior of the model and searches for patterns in the set of predictions made by the model. • Instance-level: Instance-level or local explainers [59, 30,40,62,18,61,29,43,27,6,1,50] provide explanations for specific predictions made by a model. ...
Numerous explainability methods have been proposed to shed light on the inner workings of GNNs. Despite the inclusion of empirical evaluations in all the proposed algorithms, the interrogative aspects of these evaluations lack diversity. As a result, various facets of explainability pertaining to GNNs, such as a comparative analysis of counterfactual reasoners, their stability to variational factors such as different GNN architectures, noise, stochasticity in non-convex loss surfaces, feasibility amidst domain constraints, and so forth, have yet to be formally investigated. Motivated by this need, we present a benchmarking study on perturbation-based explainability methods for GNNs, aiming to systematically evaluate and compare a wide range of explainability techniques. Among the key findings of our study, we identify the Pareto-optimal methods that exhibit superior efficacy and stability in the presence of noise. Nonetheless, our study reveals that all algorithms are affected by stability issues when faced with noisy data. Furthermore , we have established that the current generation of counterfactual explainers often fails to provide feasible recourses due to violations of topological constraints encoded by domain-specific considerations. Overall, this benchmarking study empowers stakeholders in the field of GNNs with a comprehensive understanding of the state-of-the-art explainability methods, potential research problems for further enhancement, and the implications of their application in real-world scenarios.
... The adjacency matrix, on the other hand, provides excellent interpretability by graphically depicting node connections, making it possible to comprehend network interconnections and structure with clarity [28]. Given that they learn intricate node and edge properties, which may call for more in-depth research to properly interpret, GNNs exhibit intermediate interpretability [7], [29]- [34]. High interpretability is achieved using network-based visualization, which makes it simple to identify important network properties by providing a clear visual understanding of network topology and patterns [35]. ...
There has been an increased demand for structured data mining. Graphs are among the most extensively researched data structures in discrete mathematics and computer science. Thus, it should come as no surprise that graph-based data mining has gained popularity in recent years. Graph-based methods for a transaction database are necessary to transform all the information into a graph form to conveniently extract more valuable information to improve the decision-making process. Graph-based data mining can reveal and measure process insights in a detailed structural comparison strategy that is ready for further analysis without the loss of significant details. This paper analyzes the similarities and differences among four of the most popular graph-based methods that is applied to mine rules from transaction databases by abstracting them out as a concrete high-level interface and connecting them into a common space.
... This allows for a more intuitive understanding of how information flows through the graph and it has been used in explainable methods to identify which subgraphs are important for prediction [26]. GNNs have also been shown to act as concept detectors, exhibiting strong alignment with concepts formulated as logical compositions of node degree and neighborhood properties [27]. ...
School curricula guide the daily learning activities of millions of students. They embody the understanding of the education experts who designed them of how to organize the knowledge that students should acquire in a way that is optimal for learning. This can be viewed as a learning 'theory' which is, nevertheless, rarely put to the test. Here, we model a data set obtained from a Computer-Based Formative Assessment system used by thousands of students. The student-item response matrix is highly sparse and admits a natural representation as a bipartite graph, in which nodes stand for students or items and an edge between a student and an item represents a response of the student to that item. To predict unobserved edge labels (correct/incorrect responses) we resort to a graph neural network (GNN), a machine learning method for graph-structured data. Nodes and edges are represented as multidimensional embeddings. After fitting the model, the learned item embeddings reflect properties of the curriculum, such as item difficulty and the structure of school subject domains and competences. Simulations show that the GNN is particularly advantageous over a classical model when group patterns are present in the connections between students and items, such that students from a particular group have a higher probability of successfully answering items from a specific set. In sum, important aspects of the structure of the school curriculum are reflected in response patterns from educational assessments and can be partially retrieved by our graph-based neural model.
... Explainability techniques for GNN models have been extensively studied in recent years [12][13][14][15][16][17][18][19][20][21][22][23][24][25][26]. Most existing instance-level explanation methods [12][13][14][15] provide local insights for a specific instance by extracting the features that play crucial roles in the decision procedure. ...
Model-level Graph Neural Network (GNN) explanation methods have become essential for understanding the decision-making processes of GNN models on a global scale. Many existing model-level GNN explanation methods often fail to incorporate prior knowledge of the original dataset into the initial explanation state, potentially leading to suboptimal explanation results that diverge from the real distribution of the original data. Moreover, these explainers often treat the nodes and edges within the explanation as independent elements, ignoring the structural relationships between them. This is particularly problematic in graph-based explanation tasks that are highly sensitive to structural information, which may unconsciously make the explanations miss key patterns important for the GNNs’ prediction. In this paper, we introduce KnowGNN, a knowledge-aware and structure-sensitive model-level GNN explanation framework, to explain GNN models in a global view. KnowGNN starts with a seed graph that incorporates prior knowledge of the dataset, ensuring that the final explanations accurately reflect the real data distribution. Furthermore, we construct a structure-sensitive edge mask learning method to refine the explanation process, enhancing the explanations’ ability to capture key features. Finally, we employ a simulated annealing (SA)-based strategy to control the explanation errors efficiently and thus find better explanations. We conduct extensive experiments on four public benchmark datasets. The results show that our method outperforms state-of-the-art explanation approaches by focusing explanations more closely on the actual characteristics of the data.
... Then, CBM outputs the predicted label and provides corresponding explanations via the concepts layer. Also, CBM offers ante-hoc explanations for the model's predictions due to its end-to-end training regime and has been widely applied in various fields, including healthcare [19], shift detection [20], and algorithmic reasoning [21]. ...
The Concept Bottleneck Model (CBM) is an interpretable neural network that leverages high-level concepts to explain model decisions and conduct human-machine interaction. However, in real-world scenarios, the deficiency of informative concepts can impede the model's interpretability and subsequent interventions. This paper proves that insufficient concept information can lead to an inherent dilemma of concept and label distortions in CBM. To address this challenge, we propose the Decoupling Concept Bottleneck Model (DCBM), which comprises two phases: 1) DCBM for prediction and interpretation, which decouples heterogeneous information into explicit and implicit concepts while maintaining high label and concept accuracy, and 2) DCBM for human-machine interaction, which automatically corrects labels and traces wrong concepts via mutual information estimation. The construction of the interaction system can be formulated as a light min-max optimization problem. Extensive experiments expose the success of alleviating concept/label distortions, especially when concepts are insufficient. In particular, we propose the Concept Contribution Score (CCS) to quantify the interpretability of DCBM. Numerical results demonstrate that CCS can be guaranteed by the Jensen-Shannon divergence constraint in DCBM. Moreover, DCBM expresses two effective human-machine interactions, including forward intervention and backward rectification, to further promote concept/label accuracy via interaction with human experts.
... However, the operations in vector space and tensor space are not intuitive, while the training process is deeply hidden in the black-boxes of neural models. The interpretability of such reasoning is weaker than the symbolic-based parallel reasoning [1,27,47]. ...
... In the field of eXplainable AI (XAI), efforts have historically transitioned from Local explanation to Global explanation to Mechanistic Interpretability. While local explanation methods including Selvaraju et al. (2016); Montavon et al. (2017); Sundararajan et al. (2017); Han et al. (2024) have focused on explaining specific decisions for individual instances, global explanation methods seek to uncover overall patterns and behaviors applicable across the entire dataset (Wu et al., 2022;Xuanyuan et al., 2023;Singh et al., 2024). One step further, mechanistic interpretability methods seek to analyze the fundamental components of the models and provide a holistic explanation of operational mechanics across various layers. ...
In the field of eXplainable AI (XAI) in language models, the progression from local explanations of individual decisions to global explanations with high-level concepts has laid the groundwork for mechanistic interpretability, which aims to decode the exact operations. However, this paradigm has not been adequately explored in image models, where existing methods have primarily focused on class-specific interpretations. This paper introduces a novel approach to systematically trace the entire pathway from input through all intermediate layers to the final output within the whole dataset. We utilize Pointwise Feature Vectors (PFVs) and Effective Receptive Fields (ERFs) to decompose model embeddings into interpretable Concept Vectors. Then, we calculate the relevance between concept vectors with our Generalized Integrated Gradients (GIG), enabling a comprehensive, dataset-wide analysis of model behavior. We validate our method of concept extraction and concept attribution in both qualitative and quantitative evaluations. Our approach advances the understanding of semantic significance within image models, offering a holistic view of their operational mechanics.
... For example, the black-box nature of the deep learning models can make it difficult to explain the decision-making processes and can be concerning. However, there are more than 10 methods for GNN model interpretability, including both local-and globallevel explainability [92][93][94], which can support understanding of the decision-making process within the model. Another drawback with data-driven methods, such as the neural networks, is their data hungriness. ...
Practical applications of graph neural networks (GNNs) in transportation are still a niche field. There exists a significant overlap between the potential of GNNs and the issues in strategic transport modelling. However, it is not clear whether GNN surrogates can overcome (some of) the prevalent issues. Investigation of such a surrogate will show their advantages and the disadvantages, especially throwing light on their potential to replace complex transport modelling approaches in the future, such as the agent‐based models. In this direction, as a pioneer work, this paper studies the plausibility of developing a GNN surrogate for the classical four‐step approach, one of the established strategic transport modelling approaches. A formal definition of the surrogate is presented, and an augmented data generation procedure is introduced. The network of the Greater Munich metropolitan region is used for the necessary data generation. The experimental results show that GNNs have the potential to act as transport planning surrogates and the deeper GNNs perform better than their shallow counterparts. Nevertheless, as expected, they suffer performance degradation with an increase in network size. Future research should dive deeper into formulating new GNN approaches, which are able to generalize to arbitrary large networks.
... Enhancing the interpretability of GNNs is a key factor in improving model reliability. References [61][62][63] has addressed this issue by providing transparent and interpretable decision-making processes, allowing users and developers to understand the basis of the model's recommendations, thereby increasing trust in the model. Moreover, improving the ability of GNNs to handle data noise and outliers is equally important. ...
Amid the rise of mobile technologies and Location-Based Social Networks (LBSNs), there’s an escalating demand for personalized Point-of-Interest (POI) recommendations. Especially pivotal in smart cities, these systems aim to enhance user experiences by offering location recommendations tailored to past check-ins and visited POIs. Distinguishing itself from traditional POI recommendations, the next POI approach emphasizes predicting the immediate subsequent location, factoring in both geographical attributes and temporal patterns. This approach, while promising, faces with challenges like capturing evolving user preferences and navigating data biases. The introduction of Graph Neural Networks (GNNs) brings forth a transformative solution, particularly in their ability to capture high-order dependencies between POIs, understanding deeper relationships and patterns beyond immediate connections. This survey presents a comprehensive exploration of GNN-based next POI recommendation approaches, delving into their unique characteristics, inherent challenges, and potential avenues for future research.
... In addition to those focusing on utility (e.g., F1-score in node classification tasks), a few existing studies also explored efficiency, such as comparisons on training time [60] and memory usage [32]. On the other hand, trustworthiness-oriented ones mainly aim to provide comprehensive analysis on how well graph learning models can be trusted, such as studies from the perspective of robustness [6,90] and interpretability [2,77]. However, from the perspective of algorithmic fairness, existing benchmarks remain scarce. ...
Fairness-aware graph learning has gained increasing attention in recent years. Nevertheless, there lacks a comprehensive benchmark to evaluate and compare different fairness-aware graph learning methods, which blocks practitioners from choosing appropriate ones for broader real-world applications. In this paper, we present an extensive benchmark on ten representative fairness-aware graph learning methods. Specifically, we design a systematic evaluation protocol and conduct experiments on seven real-world datasets to evaluate these methods from multiple perspectives, including group fairness, individual fairness, the balance between different fairness criteria, and computational efficiency. Our in-depth analysis reveals key insights into the strengths and limitations of existing methods. Additionally, we provide practical guidance for applying fairness-aware graph learning methods in applications. To the best of our knowledge, this work serves as an initial step towards comprehensively understanding representative fairness-aware graph learning methods to facilitate future advancements in this area.
... Additional works face the explainability problem from different perspectives as explanation supervision (Gao et al., 2021), neuron analysis (Xuanyuan et al., 2023), and motifbased generation (Yu & Gao, 2022). For a comprehensive discussion on methods to explain GNNs, we refer the reader to the survey . ...
Graph Neural Networks (GNNs) are a popular class of machine learning models. Inspired by the learning to explain (L2X) paradigm, we propose L2xGnn, a framework for explainable GNNs which provides faithful explanations by design. L2xGnn learns a mechanism for selecting explanatory subgraphs (motifs) which are exclusively used in the GNNs message-passing operations. L2xGnn is able to select, for each input graph, a subgraph with specific properties such as being sparse and connected. Imposing such constraints on the motifs often leads to more interpretable and effective explanations. Experiments on several datasets suggest that L2xGnn achieves the same classification accuracy as baseline methods using the entire input graph while ensuring that only the provided explanations are used to make predictions. Moreover, we show that L2xGnn is able to identify motifs responsible for the graph’s properties it is intended to predict.
... A concept and its importance are represented by a cluster and the number of nodes in it respectively. Another method GCneuron [113], which is inspired by Compositional Explanations of Neurons [71], finds global explanation for GNNs by finding compositional concepts aligned with neurons. A base concept is a function C on Graph G that produces a binary mask over all the input nodes V . ...
Graph neural networks (GNNs) are powerful graph-based deep-learning models that have gained significant attention and demonstrated remarkable performance in various domains, including natural language processing, drug discovery, and recommendation systems. However, combining feature information and combinatorial graph structures has led to complex non-linear GNN models. Consequently, this has increased the challenges of understanding the workings of GNNs and the underlying reasons behind their predictions. To address this, numerous explainability methods have been proposed to shed light on the inner mechanism of the GNNs. Explainable GNNs improve their security and enhance trust in their recommendations. This survey aims to provide a comprehensive overview of the existing explainability techniques for GNNs. We create a novel taxonomy and hierarchy to categorize these methods based on their objective and methodology. We also discuss the strengths, limitations, and application scenarios of each category. Furthermore, we highlight the key evaluation metrics and datasets commonly used to assess the explainability of GNNs. This survey aims to assist researchers and practitioners in understanding the existing landscape of explainability methods, identifying gaps, and fostering further advancements in interpretable graph-based machine learning.
... There already exists work that evaluates GNNs w.r.t. learning concepts [26][27][28][29][30][31]. However, these works do not evaluate existing relevance-based GNN explainers regarding the fit between the relevance they compute and the features that represent important sub-concepts from an application domain's and user's perspective. ...
Graph Neural Networks (GNN) show good performance in relational data classification. However, their contribution to concept learning and the validation of their output from an application domain’s and user’s perspective have not been thoroughly studied. We argue that combining symbolic learning methods, such as Inductive Logic Programming (ILP), with statistical machine learning methods, especially GNNs, is an essential forward-looking step to perform powerful and validatable relational concept learning. In this contribution, we introduce a benchmark for the conceptual validation of GNN classification outputs. It consists of the symbolic representations of symmetric and non-symmetric figures that are taken from a well-known Kandinsky Pattern data set. We further provide a novel validation framework that can be used to generate comprehensible explanations with ILP on top of the relevance output of GNN explainers and human-expected relevance for concepts learned by GNNs. Our experiments conducted on our benchmark data set demonstrate that it is possible to extract symbolic concepts from the most relevant explanations that are representative of what a GNN has learned. Our findings open up a variety of avenues for future research on validatable explanations for GNNs.
The increasing use of black-box networks in high-risk contexts has led researchers to propose explainable methods to make these networks transparent. Most methods that allow us to understand the behavior of Deep Neural Networks (DNNs) are post-hoc approaches, implying that the explainability is questionable, as these methods do not clarify the internal behavior of a model. Thus, this demonstrates the difficulty of interpreting the internal behavior of deep models. This systematic literature review collects the ante-hoc methods that provide an understanding of the internal mechanisms of deep models and which can be helpful to researchers who need to use interpretability methods to clarify DNNs. This work provides the definitions of strong interpretability and weak interpretability, which will be used to describe the interpretability of the methods discussed in this paper. The results of this work are divided mainly into prototype-based methods, concept-based methods, and other interpretability methods for deep models.
To address the barrier caused by the black-box nature of Deep Learning (DL) for practical deployment, eXplainable Artificial Intelligence (XAI) has emerged and is developing rapidly. While significant progress has been made in explanation techniques for DL models targeted to images and texts, research on explaining DL models for graph data is still in its infancy. As Graph Neural Networks (GNNs) have shown superiority over various network analysis tasks, their explainability has also gained attention from both academia and industry. However, despite the increasing number of GNN explanation methods, there is currently neither a fine-grained taxonomy of them, nor a holistic set of evaluation criteria for quantitative and qualitative evaluation. To fill this gap, we conduct a comprehensive survey on existing explanation methods of GNNs in this paper. Specifically, we propose a novel four-dimensional taxonomy of GNN explanation methods and summarize evaluation criteria in terms of correctness, robustness, usability, understandability, and computational complexity. Based on the taxonomy and criteria, we thoroughly review the recent advances in GNN explanation methods and analyze their pros and cons. In the end, we identify a series of open issues and put forward future research directions to facilitate XAI research in the field of GNNs.
Graph learning models achieve state-of-the-art performance on many tasks, but this often requires increasingly large model sizes. Accordingly, the complexity of their representations increase. Explainability techniques (XAI) have made remarkable progress in the interpretability of ML models. However, the non-relational nature of Graph Neural Networks (GNNs) make it difficult to reuse already existing XAI methods. While other works have focused on instance-based explanation methods for GNNs, very few have investigated model-based methods and, to our knowledge, none have tried to probe the embedding of the GNNs for well-known structural graph properties. In this paper we present a model agnostic explainability pipeline for Graph Neural Networks (GNNs) employing diagnostic classifiers. This pipeline aims to probe and interpret the learned representations in GNNs across various architectures and datasets, refining our understanding and trust in these models.
Numerous explainability techniques have been developed to reveal the prediction principles of Graph Neural Networks (GNNs) across diverse domains. However, many existing approaches, particularly those concentrating on model-level explanations, tend to grapple with the tunnel vision problem, leading to less-than-optimal outcomes and constraining users' comprehensive understanding of GNNs. Furthermore, these methods typically require hyperparameters to mold the explanations, introducing unintended human biases. In response, we present GAXG, a global and self-adaptive optimal graph topology generation framework for explaining GNNs' prediction principles at model-level. GAXG addresses the challenges of tunnel vision and hyperparameter reliance by integrating a strategically tailored Monte Carlo Tree Search (MCTS) algorithm. Notably, our tailored MCTS algorithm is modified to incorporate an Edge Mask Learning and Simulated Annealing-based subgraph screening strategy during the expansion phase, resolving the inherent time-consuming challenges of the tailored MCTS and enhancing the quality of the generated explanatory graph topologies. Experimental results underscore GAXG's effectiveness in discovering global explanations for GNNs, outperforming leading explainers on most evaluation metrics.
Counterfactual subgraphs explain Graph Neural Networks (GNNs) by answering the question: “How would the prediction change if a certain subgraph were absent in the input instance?” The differentiable proxy adjacency matrix is prevalent in current counterfactual subgraph discovery studies due to its ability to avoid exhaustive edge searching. However, a prediction gap exists when feeding the proxy matrix with continuous values and the thresholded discrete adjacency matrix to GNNs, compromising the optimization of the subgraph generator. Furthermore, the end-to-end learning schema adopted in the subgraph generator limits the diversity of counterfactual subgraphs. To this end, we propose CF-GFNExplainer, a flow-based approach for learning counterfactual subgraphs. CF-GFNExplainer employs a policy network with a discrete edge removal schema to construct counterfactual subgraph generation trajectories. Additionally, we introduce a loss function designed to guide CF-GFNExplainer’s optimization. The discrete adjacency matrix generated in each trajectory eliminates the prediction gap, enhancing the validity of the learned subgraphs. Furthermore, the multi-trajectories sampling strategy adopted in CF-GFNExplainer results in diverse counterfactual subgraphs. Extensive experiments conducted on synthetic and real-world datasets demonstrate the effectiveness of the proposed method in terms of validity and diversity. The data and code of CF-GFNExplainer are available
1
1
https://github.com/AmGracee/CF-GFNExplainer
.
Generating explanations for graph neural networks (GNNs) has been studied to understand their behaviors in analytical tasks such as graph classification. Existing approaches aim to understand the overall results of GNNs rather than providing explanations for specific class labels of interest, and may return explanation structures that are hard to access, nor directly queryable. We propose GVEX, a novel paradigm that generates Graph Views for GNN EXplanation. (1) We design a two-tier explanation structure called explanation views. An explanation view consists of a set of graph patterns and a set of induced explanation subgraphs. Given a database G of multiple graphs and a specific class label l assigned by a GNN-based classifier M, it concisely describes the fraction of G that best explains why l is assigned by M. (2) We propose quality measures and formulate an optimization problem to compute optimal explanation views for GNN explanation. We show that the problem is Σ2P-hard. (3) We present two algorithms. The first one follows an explain-and-summarize strategy that first generates high-quality explanation subgraphs which best explain GNNs in terms of feature influence maximization, and then performs a summarization step to generate patterns. We show that this strategy provides an approximation ratio of 1/2. Our second algorithm performs a single-pass to an input node stream in batches to incrementally maintain explanation views, having an anytime quality guarantee of 1/4-approximation. Using real-world benchmark data, we experimentally demonstrate the effectiveness, efficiency, and scalability of GVEX. Through case studies, we showcase the practical applications of GVEX.
Algorithmic decision support systems are widely applied in domains ranging from healthcare to journalism. To ensure that these systems are fair and accountable, it is essential that humans can maintain meaningful agency, understand and oversee algorithmic processes. Explainability is often seen as a promising mechanism for enabling human-in-the-loop, however, current approaches are ineffective and can lead to various biases. We argue that explainability should be tailored to support naturalistic decision-making and sensemaking strategies employed by domain experts and novices. Based on cognitive psychology and human factors literature review we map potential decision-making strategies dependent on expertise, risk and time dynamics and propose the conceptual Expertise, Risk and Time Explainability Framework, intended to be used as explainability design guidelines. Finally, we present a worked example in journalism to illustrate the applicability of our framework in practice.
As graph neural networks are becoming more and more powerful and useful in the field of drug discovery, many pharmaceutical companies are getting interested in utilizing these methods for their own in-house frameworks. This is especially compelling for tasks such as the prediction of molecular properties which is often one of the most crucial tasks in computer-aided drug discovery workflows. The immense hype surrounding these kinds of algorithms has led to the development of many different types of promising architectures and in this review we try to structure this highly dynamic field of AI-research by collecting and classifying 80 GNNs that have been used to predict more than 20 molecular properties using 48 different datasets.
Background
Explainability is one of the most heavily debated topics when it comes to the application of artificial intelligence (AI) in healthcare. Even though AI-driven systems have been shown to outperform humans in certain analytical tasks, the lack of explainability continues to spark criticism. Yet, explainability is not a purely technological issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration. This paper provides a comprehensive assessment of the role of explainability in medical AI and makes an ethical evaluation of what explainability means for the adoption of AI-driven tools into clinical practice.
Methods
Taking AI-based clinical decision support systems as a case in point, we adopted a multidisciplinary approach to analyze the relevance of explainability for medical AI from the technological, legal, medical, and patient perspectives. Drawing on the findings of this conceptual analysis, we then conducted an ethical assessment using the “Principles of Biomedical Ethics” by Beauchamp and Childress (autonomy, beneficence, nonmaleficence, and justice) as an analytical framework to determine the need for explainability in medical AI.
Results
Each of the domains highlights a different set of core considerations and values that are relevant for understanding the role of explainability in clinical practice. From the technological point of view, explainability has to be considered both in terms how it can be achieved and what is beneficial from a development perspective. When looking at the legal perspective we identified informed consent, certification and approval as medical devices, and liability as core touchpoints for explainability. Both the medical and patient perspectives emphasize the importance of considering the interplay between human actors and medical AI. We conclude that omitting explainability in clinical decision support systems poses a threat to core ethical values in medicine and may have detrimental consequences for individual and public health.
Conclusions
To ensure that medical AI lives up to its promises, there is a need to sensitize developers, healthcare professionals, and legislators to the challenges and limitations of opaque algorithms in medical AI and to foster multidisciplinary collaboration moving forward.
Tree-based machine learning models such as random forests, decision trees and gradient boosted trees are popular nonlinear predictive models, yet comparatively little attention has been paid to explaining their predictions. Here we improve the interpretability of tree-based models through three main contributions. (1) A polynomial time algorithm to compute optimal explanations based on game theory. (2) A new type of explanation that directly measures local feature interaction effects. (3) A new set of tools for understanding global model structure based on combining many local explanations of each prediction. We apply these tools to three medical machine learning problems and show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. These tools enable us to (1) identify high-magnitude but low-frequency nonlinear mortality risk factors in the US population, (2) highlight distinct population subgroups with shared risk characteristics, (3) identify nonlinear interaction effects among risk factors for chronic kidney disease and (4) monitor a machine learning model deployed in a hospital by identifying which features are degrading the model’s performance over time. Given the popularity of tree-based machine learning models, these improvements to their interpretability have implications across a broad set of domains. Tree-based machine learning models are widely used in domains such as healthcare, finance and public services. The authors present an explanation method for trees that enables the computation of optimal local explanations for individual predictions, and demonstrate their method on three medical datasets.
We propose a technique for producing ‘visual explanations’ for decisions from a large class of Convolutional Neural Network (CNN)-based models, making them more transparent and explainable. Our approach—Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’ in a classification network or a sequence of words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept. Unlike previous approaches, Grad-CAM is applicable to a wide variety of CNN model-families: (1) CNNs with fully-connected layers (e.g.VGG), (2) CNNs used for structured outputs (e.g.captioning), (3) CNNs used in tasks with multi-modal inputs (e.g.visual question answering) or reinforcement learning, all without architectural changes or re-training. We combine Grad-CAM with existing fine-grained visualizations to create a high-resolution class-discriminative visualization, Guided Grad-CAM, and apply it to image classification, image captioning, and visual question answering (VQA) models, including ResNet-based architectures. In the context of image classification models, our visualizations (a) lend insights into failure modes of these models (showing that seemingly unreasonable predictions have reasonable explanations), (b) outperform previous methods on the ILSVRC-15 weakly-supervised localization task, (c) are robust to adversarial perturbations, (d) are more faithful to the underlying model, and (e) help achieve model generalization by identifying dataset bias. For image captioning and VQA, our visualizations show that even non-attention based models learn to localize discriminative regions of input image. We devise a way to identify important neurons through Grad-CAM and combine it with neuron names (Bau et al. in Computer vision and pattern recognition, 2017) to provide textual explanations for model decisions. Finally, we design and conduct human studies to measure if Grad-CAM explanations help users establish appropriate trust in predictions from deep networks and show that Grad-CAM helps untrained users successfully discern a ‘stronger’ deep network from a ‘weaker’ one even when both make identical predictions. Our code is available at https://github.com/ramprs/grad-cam/, along with a demo on CloudCV (Agrawal et al., in: Mobile cloud visual media computing, pp 265–290. Springer, 2015) (http://gradcam.cloudcv.org) and a video at http://youtu.be/COjUB9Izk6E.
Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. Previous work largely focused on what these models learn at the representation level. We break this analysis down further and study individual dimensions (neu-rons) in the vector representation learned by end-to-end neu-ral models in NLP tasks. We propose two methods: Linguistic Correlation Analysis, based on a supervised method to extract the most relevant neurons with respect to an extrin-sic task, and Cross-model Correlation Analysis, an unsuper-vised method to extract salient neurons w.r.t. the model itself. We evaluate the effectiveness of our techniques by ablating the identified neurons and reevaluating the network's performance for two tasks: neural machine translation (NMT) and neural language modeling (NLM). We further present a comprehensive analysis of neurons with the aim to address the following questions: i) how localized or distributed are different linguistic properties in the models? ii) are certain neu-rons exclusive to some properties and not others? iii) is the information more or less distributed in NMT vs. NLM? and iv) how important are the neurons identified through the linguistic correlation method to the overall task? Our code is publicly available 1 as part of the NeuroX toolkit (Dalvi et al. 2019).
This tutorial provides a gentle introduction to kernel density estimation (KDE) and recent advances regarding confidence bands and geometric/topological features. We begin with a discussion of basic properties of KDE: the convergence rate under various metrics, density derivative estimation, and bandwidth selection. Then, we introduce common approaches to the construction of confidence intervals/bands, and we discuss how to handle bias. Next, we talk about recent advances in the inference of geometric and topological features of a density function using KDE. Finally, we illustrate how one can use KDE to estimate a cumulative distribution function and a receiver operating characteristic curve. We provide R implementations related to this tutorial at the end.
Computational approaches to protein function prediction infer protein function by finding proteins with similar sequence, structure, surface clefts, chemical properties, amino acid motifs, interaction partners or phylogenetic profiles. We present a new approach that combines sequential, structural and chemical information into one graph model of proteins. We predict functional class membership of enzymes and non-enzymes using graph kernels and support vector machine classification on these protein graphs.
Our graph model, derivable from protein sequence and structure only, is competitive with vector models that require additional protein information, such as the size of surface pockets. If we include this extra information into our graph model, our classifier yields significantly higher accuracy levels than the vector models. Hyperkernels allow us to select and to optimally combine the most relevant node attributes in our protein graphs. We have laid the foundation for a protein function prediction system that integrates protein information from various sources efficiently and effectively.
More information available via www.dbs.ifi.lmu.de/Mitarbeiter/borgwardt.html.
Graph Neural Networks (GNNs) are a powerful tool for machine learning on graphs. GNNs combine node feature information with the graph structure by recursively passing neural messages along edges of the input graph. However, incorporating both graph structure and feature information leads to complex models and explaining predictions made by GNNs remains unsolved. Here we propose GnnExplainer, the first general, model-agnostic approach for providing interpretable explanations for predictions of any GNN-based model on any graph-based machine learning task. Given an instance, GnnExplainer identifies a compact subgraph structure and a small subset of node features that have a crucial role in GNN's prediction. Further, GnnExplainer can generate consistent and concise explanations for an entire class of instances. We formulate GnnExplainer as an optimization task that maximizes the mutual information between a GNN's prediction and distribution of possible subgraph structures. Experiments on synthetic and real-world graphs show that our approach can identify important graph structures as well as node features, and outperforms alternative baseline approaches by up to 43.0% in explanation accuracy. GnnExplainer provides a variety of benefits, from the ability to visualize semantically relevant structures to interpretability, to giving insights into errors of faulty GNNs.
In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network to have remarkable localization ability despite being trained on image-level labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that can be applied to a variety of tasks. Despite the apparent simplicity of global average pooling, we are able to achieve 37.1% top-5 error for object localization on ILSVRC 2014, which is remarkably close to the 34.2% top-5 error achieved by a fully supervised CNN approach. We demonstrate that our network is able to localize the discriminative image regions on a variety of tasks despite not being trained for them
A review of the literature yielded data on over 200 aromatic and heteroaromatic nitro compounds tested for mutagenicity in the Ames test using S. typhimurium TA98. From the data, a quantitative structure-activity relationship (QSAR) has been derived for 188 congeners. The main determinants of mutagenicity are the hydrophobicity (modeled by octanol/water partition coefficients) and the energies of the lowest unoccupied molecular orbitals calculated using the AM1 method. It is also shown that chemicals possessing three or more fused rings possess much greater mutagenic potency than compounds with one or two fused rings. Since the QSAR is based on a very wide range in structural variation, aromatic rings from benzene to coronene are included as well as many different types of heterocycles, it is a significant step toward a predictive toxicology of value in the design of less mutagenic bioactive compounds.
Mutagenicity is one of the numerous adverse properties of a compound that hampers its potential to become a marketable drug. Toxic properties can often be related to chemical structure, more specifically, to particular substructures, which are generally identified as toxicophores. A number of toxicophores have already been identified in the literature. This study aims at increasing the current degree of reliability and accuracy of mutagenicity predictions by identifying novel toxicophores from the application of new criteria for toxicophore rule derivation and validation to a considerably sized mutagenicity dataset. For this purpose, a dataset of 4337 molecular structures with corresponding Ames test data (2401 mutagens and 1936 nonmutagens) was constructed. An initial substructure-search of this dataset showed that most mutagens were detected by applying only eight general toxicophores. From these eight, more specific toxicophores were derived and approved by employing chemical and mechanistic knowledge in combination with statistical criteria. A final set of 29 toxicophores containing new substructures was assembled that could classify the mutagenicity of the investigated dataset with a total classification error of 18%. Furthermore, mutagenicity predictions of an independent validation set of 535 compounds were performed with an error percentage of 15%. Since these error percentages approach the average interlaboratory reproducibility error of Ames tests, which is 15%, it was concluded that these toxicophores can be applied to risk assessment processes and can guide the design of chemical libraries for hit and lead optimization.
Entropy-based logic explanations of neural networks
- P Barbiero
- G Ciravegna
- F Giannini
- P Liò
- M Gori
- S Melacci
Barbiero, P.; Ciravegna, G.; Giannini, F.; Liò, P.; Gori, M.;
and Melacci, S. 2022. Entropy-based logic explanations of
neural networks. In Proceedings of the AAAI Conference on
Artificial Intelligence, volume 36, 6046-6054.
- E Dai
- T Zhao
- H Zhu
- J Xu
- Z Guo
- H Liu
- J Tang
- S Wang
Dai, E.; Zhao, T.; Zhu, H.; Xu, J.; Guo, Z.; Liu, H.; Tang, J.;
and Wang, S. 2022. A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness,
and Explainability. arXiv preprint arXiv:2204.08570.
The Judicial Demand for Explainable Artificial Intelligence
- A Deeks
Deeks, A. 2019. The Judicial Demand for Explainable Artificial Intelligence. Columbia Law Review, 119(7): 1829-1850.
Algorithmic Concept-based Explainable Reasoning
- D Georgiev
- P Barbiero
- D Kazhdan
- P Veličković
- P Liò
Georgiev, D.; Barbiero, P.; Kazhdan, D.; Veličković, P.; and
Liò, P. 2022. Algorithmic Concept-based Explainable Reasoning. In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 36, 6685-6693.
Towards Automatic Concept-Based Explanations
- A Ghorbani
- J Wexler
- J Y Zou
- B Kim
Ghorbani, A.; Wexler, J.; Zou, J. Y.; and Kim, B. 2019. Towards Automatic Concept-Based Explanations. Advances in
Neural Information Processing Systems, 32.
A New Concept for Explaining Graph Neural Networks
- A Himmelhuber
- S Zillner
- S Grimm
- M Ringsquandl
- M Joblin
- T A Runkler
Himmelhuber, A.; Zillner, S.; Grimm, S.; Ringsquandl, M.;
Joblin, M.; and Runkler, T. A. 2021. A New Concept for
Explaining Graph Neural Networks. In NeSy, 1-5.
Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)
- B Kim
- M Wattenberg
- J Gilmer
- C Cai
- J Wexler
- F Viegas
Kim, B.; Wattenberg, M.; Gilmer, J.; Cai, C.; Wexler, J.;
Viegas, F.; et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors
(tcav). In International Conference on Machine Learning,
2668-2677. PMLR.
Concept Bottleneck Models
- P W Koh
- T Nguyen
- Y S Tang
- S Mussmann
- E Pierson
- B Kim
- P Liang
Koh, P. W.; Nguyen, T.; Tang, Y. S.; Mussmann, S.; Pierson, E.; Kim, B.; and Liang, P. 2020. Concept Bottleneck
Models. In International Conference on Machine Learning,
5338-5348. PMLR.
- J Li
- W Monroe
- D Jurafsky
Li, J.; Monroe, W.; and Jurafsky, D. 2016. Understanding
Neural Networks Through Representation Erasure. arXiv
preprint arXiv:1612.08220.
Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
- Q Li
- Z Han
- X.-M Wu
Li, Q.; Han, Z.; and Wu, X.-M. 2018. Deeper Insights into
Graph Convolutional Networks for Semi-Supervised Learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
DIG: A Turnkey Library for Diving into Graph Deep Learning Research
- M Liu
- Y Luo
- L Wang
- Y Xie
- H Yuan
- S Gui
- H Yu
- Z Xu
- J Zhang
- Y Liu
Liu, M.; Luo, Y.; Wang, L.; Xie, Y.; Yuan, H.; Gui, S.;
Yu, H.; Xu, Z.; Zhang, J.; Liu, Y.; et al. 2021. DIG: A
Turnkey Library for Diving into Graph Deep Learning Research. Journal of Machine Learning Research, 22(240):
1-9.
Parameterized Explainer for Graph Neural Network
- D Luo
- W Cheng
- D Xu
- W Yu
- B Zong
- H Chen
- X Zhang
Luo, D.; Cheng, W.; Xu, D.; Yu, W.; Zong, B.; Chen, H.; and
Zhang, X. 2020. Parameterized Explainer for Graph Neural
Network. Advances in Neural Information Processing Systems, 33.
- L C Magister
- D Kazhdan
- V Singh
- P Liò
Magister, L. C.; Kazhdan, D.; Singh, V.; and Liò, P.
2021. GCExplainer: Human-in-the-Loop Concept-based
Explanations for Graph Neural Networks. arXiv preprint
arXiv:2107.11889.
Compositional Explanations of Neurons
- J Mu
Mu, J.; and Andreas, J. 2020. Compositional Explanations
of Neurons. Advances in Neural Information Processing
Systems, 33: 17153-17163.
Learning to Simulate Complex Physics with Graph Networks
- A Sanchez-Gonzalez
- J Godwin
- T Pfaff
- R Ying
- J Leskovec
- P Battaglia
Sanchez-Gonzalez, A.; Godwin, J.; Pfaff, T.; Ying, R.;
Leskovec, J.; and Battaglia, P. 2020. Learning to Simulate
Complex Physics with Graph Networks. In International
Conference on Machine Learning, 8459-8468. PMLR.
Opening the Black Box of Deep Neural Networks via Information
- R Shwartz-Ziv
- N Tishby
Shwartz-Ziv, R.; and Tishby, N. 2017. Opening the Black
Box of Deep Neural Networks via Information. arXiv
preprint arXiv:1703.00810.
- P Veličković
Veličković, P. 2022. Message Passing All the Way Up. arXiv
preprint arXiv:2202.11097.
PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks
- M Vu
- M T Thai
Vu, M.; and Thai, M. T. 2020. PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks. Advances in Neural Information Processing Systems,
33: 12225-12235.
Semi-Supervised Classification with Graph Convolutional Networks
- M Welling
- T N Kipf
Welling, M.; and Kipf, T. N. 2016. Semi-Supervised Classification with Graph Convolutional Networks. In J. International Conference on Learning Representations (ICLR
2017).
- L Wu
- Y Chen
- K Shen
- X Guo
- H Gao
- S Li
- J Pei
- B Long
Wu, L.; Chen, Y.; Shen, K.; Guo, X.; Gao, H.; Li, S.;
Pei, J.; and Long, B. 2021. Graph Neural Networks for
Natural Language Processing: A Survey. arXiv preprint
arXiv:2106.06090.
How Powerful are Graph Neural Networks? ArXiv
- K Xu
- W Hu
- J Leskovec
- S Jegelka
Xu, K.; Hu, W.; Leskovec, J.; and Jegelka, S. 2019.
How Powerful are Graph Neural Networks?
ArXiv,
abs/1810.00826.