No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Interpretability Index Based on Balanced Volumes for Transparent Models and Agnostic Explainers
Abstract
We discuss interpretability and explainability of machine learning models. We introduce a universal interpretability index, JJ, to quantify and monitor the interpretability of a general-purpose model, which can be static or evolve incre-mentally from a data stream. The models can be transparent classifiers, predictors or controllers operating on partitions or granules of the data space, e.g., rule-based models, trees, proba-bilistic clustering models, modular or granular neural networks. Additionally, black boxes can be monitored either after the derivation of a global or local surrogate model as a result of the application of a model-agnostic explainer. The index does not de-pend on the type of algorithm that creates or updates the model, i.e., supervised, unsupervised, semi-supervised, or reinforcement. While loss or error-related indices, validity measures, processing time, and closed-loop criteria have been used to evaluate model effectiveness across different fields, a general interpretability index in consonance with explainable AI does not exist. The index JJ is computed straightforwardly. It reflects the principle of justifiable granularity by taking into account balanced volumes, number of partitions and dependent parameters, and features per partition. The index advocates that a concise model founded on balanced partitions offers a higher level of interpretability. It facilitates comparisons between models, can be used as a term in loss functions or embedded into learning procedures to motivate equilibrium of volumes, and ultimately support human-centered decision making. Computational experiments show the versatility of the index in a biomedical prediction problem from speech data, and in image classification.
... Toward the development of computational models based on voice and speech data for BD monitoring, the two most fundamental aspects are prediction accuracy and model interpretability or explainability [8] [9]. Practically speaking, interpretability and explainability are tied or very close concepts, often used interchangeably [10]. ...
... The demand for the ability to understand and trust machine learning and artificial intelligence systems has increased recently in domains such as healthcare, criminal justice, and computer vision, influenced by deep neural nets and ensembles of black box models [8]. More interpretable models and explanation methods have been developed over the past few years [9]. The next section describes some of these developments with emphasis on BD diagnosis [39] [40] [41]. ...
... The interpretability of an evolving model is intuitively higher if it is easier for a person to reason and trace back why a prediction was made. In-model interpretability concerns models that have inherent interpretability potential in them through the use of meaningful basic entities and constraints [9]. Different from in-model intrinsic interpretability potential, post-model interpretability refers to improving the interpretability degree after building the model (post hoc). ...
We apply an evolving granular-computing modeling approach, called evolving Optimal Granular System (eOGS), to bipolar mood disorder (BD) diagnosis based on speech data streams. The eOGS online learning algorithm reveals information granules in the flow and design the structure and parameters of a granular rule-based model with a certain degree of interpretability based on acoustic attributes obtained from phone calls made over 7 months to the Psychiatry department of a hospital. A multi-objective programming problem that trades-off information specificity, model compactness, and numerical and granular error indices is presented. Spectral and prosodic attributes are ranked and selected based on a hybrid Pearson-Spearman correlation coefficient. Low attribute-class correlation, ranging from 0.03 to 0.07, is observed, as well as high class overlap, which is typical in the psychiatric field. eOGS models for BD recognition overcome alternative computational-intelligence models, namely, Dynamic Evolving Neural-Fuzzy Inference System (DENFIS) and Fuzzy-set-Based evolving Modeling (FBeM-Gauss), by a small margin in both best and average cases; followed by eXtended Takagi-Sugeno (xTS) and evolving Takagi Sugeno (eTS) types of models. The proposed eOGS model using only 8 of the original acoustic attributes, and about 15 ‘If-Then’ inference rules, has exhibited the best root mean square error, 0.1361, and 91.8% accuracy in sharp BD class estimates. Granules associated to linguistic labels and a granular input-output map offer human understandability with relation to the inherent process of generating class estimates. Linguistically readable eOGS rules may assist physicians in explaining symptoms and making a diagnosis.
... As this paper explores the application of eGNN as a classifier for EEG data, the model outcomes are basically class probabilities. The overall network processing is transparent, meaning that its inherent interpretability can be quantified according to the II interpretability index [31]. ...
... The index II [31] is useful to quantify the interpretability of eGNN-C+. The index considers factors such as balanced volumes, number of granules and dependent parameters, and features per partition. ...
... The index says that a smaller set of concise rules supported by granules with balanced volumes, carrying fewer local parameters, contributes to a higher level of model understandability. Refer to [31] for a complete description. ...
We introduce a modified incremental learning algorithm for evolving Granular Neural Network Classifiers (eGNN-C+). We use double-boundary hyper-boxes to represent granules, and customize the adaptation procedures to enhance the robustness of outer boxes for data coverage and noise suppression, while ensuring that inner boxes remain flexible to capture drifts. The classifier evolves from scratch, incorporates new classes on the fly, and performs local incremental feature weighting. As an application, we focus on the classification of emotion-related patterns within electroencephalogram (EEG) signals. Emotion recognition is crucial for enhancing the realism and interactivity of computer systems. The challenge lies exactly in developing high-performance algorithms capable of effectively managing individual differences and non-stationarities in physiological data without relying on subject-specific information. We extract features from the Fourier spectrum of EEG signals obtained from 28 individuals engaged in playing computer games – a public dataset. Each game elicits a different predominant emotion: boredom, calmness, horror, or joy. We analyze individual electrodes, time window lengths, and frequency bands to assess the accuracy and interpretability of resulting user-independent neural models. The findings indicate that both brain hemispheres assist classification, especially electrodes on the temporal (T8) and parietal (P7) areas, alongside contributions from frontal and occipital electrodes. While patterns may manifest in any band, the Alpha (8-13Hz), Delta (1-4Hz), and Theta (4-8Hz) bands, in this order, exhibited higher correspondence with the emotion classes. The eGNN-C+ demonstrates effectiveness in learning EEG data. It achieves an accuracy of 81.7% and a 0.0029 II interpretability using 10-second time windows, even in face of a highly-stochastic time-varying 4-class classification problem.
... As this paper explores the application of eGNN as a classifier for EEG data, the model outcomes are basically class probabilities. The overall network processing is transparent, meaning that its inherent interpretability can be quantified according to the II interpretability index [31]. ...
... The index II [31] is useful to quantify the interpretability of eGNN-C+. The index considers factors such as balanced volumes, number of granules and dependent parameters, and features per partition. ...
... The index says that a smaller set of concise rules supported by granules with balanced volumes, carrying fewer local parameters, contributes to a higher level of model understandability. Refer to [31] for a complete description. ...
We introduce a modified incremental learning algorithm for evolving Granular Neural Network Classifiers (eGNN-C+). We use double-boundary hyper-boxes to represent granules, and customize the adaptation procedures to enhance the robust-ness of outer boxes for data coverage and noise suppression, while ensuring that inner boxes remain flexible to capture drifts. The classifier evolves from scratch, incorporates new classes on the fly, and performs local incremental feature weighting. As an application, we focus on the classification of emotion-related patterns within electroencephalogram (EEG) signals. Emotion recognition is crucial for enhancing the realism and interactivity of computer systems. The challenge lies exactly in developing high-performance algorithms capable of effectively managing individual differences and non-stationarities in physiological data without relying on subject-specific calibration data. We extract features from the Fourier spectrum of EEG signals obtained from 28 individuals engaged in playing computer games-a public dataset. Each game elicits a different predominant emotion: boredom, calmness, horror, or joy. We analyze individual electrodes, time window lengths, and frequency bands to assess the accuracy and interpretability of resulting user-independent neural models. The findings indicate that both brain hemispheres assist classification, especially electrodes on the temporal (T8) and parietal (P7) areas, alongside contributions from frontal and occipital electrodes. While patterns may manifest in any band, the Alpha (8-13Hz), Delta (1-4Hz), and Theta (4-8Hz) bands, in this order, exhibited higher correspondence with the emotion classes. The eGNN-C+ demonstrates effectiveness in learning EEG data. It achieves an accuracy of 81.7% and a 0.0029 II interpretability using 10-second time windows, even in face of a highly-stochastic time-varying 4-class classification problem.
... O uso crescente de aprendizado de máquina destaca-se especialmente na análise de grandes volumes de dados (Verne et al., 2022). Dois desafios em particular são aqueles relativos a interpretabilidade e a confiança em modelos profundos (Souza and Dragoni, 2024) (Leite et al., 2024b), e a capacidade de aprendizado online contínuo para capturar novos padrões em ambientes dinâmicos (Mai et al., 2022) (Shaheen et al., 2022) (Leite et al., 2024a). ...
Recent advances in machine learning for computer vision and image classification have presented significant challenges, such as interpretability of deep neural network models and ability for continuous learning in dynamic environments. This paper introduces Convolutional Evolving Granular Neural Networks (CEGNN). A CEGNN combines the feature extraction components of a VGG-16 convolutional net with an evolving granular neural network (EGNN) toward advancing the understanding and applications of incremental learning in computer vision, particularly in image recognition and classification. An incremental algorithm is incorporated to the CEGNN to improve model interpretability and continuous learning. Experimental results show that the CEGNN is efficient and competitive in image classification, achieving an accuracyof 78.9% and a precision of 79.0% in a 10-class problem. This opens research avenues in certain applications, such as those dealing with medical images, satellite images, and image-based closed-loop autonomous systems.
... O uso crescente de aprendizado de máquina destaca-se especialmente na análise de grandes volumes de dados (Verne et al., 2022). Dois desafios em particular são aqueles relativos a interpretabilidade e a confiança em modelos profundos (Souza and Dragoni, 2024) (Leite et al., 2024b), e a capacidade de aprendizado online contínuo para capturar novos padrões em ambientes dinâmicos (Mai et al., 2022) (Shaheen et al., 2022) (Leite et al., 2024a). ...
Recent advances in machine learning for computer vision and image classification have presented significant challenges, such as interpretability of deep neural network models and ability for continuous learning in dynamic environments. This paper introduces Convolutional Evolving Granular Neural Networks (CEGNN). A CEGNN combines the feature extraction components of a VGG-16 convolutional net with an evolving granular neural network (EGNN) toward advancing the understanding and applications of incremental learning in computer vision, particularly in image recognition and classification. An incremental algorithm is incorporated to the CEGNN to improve model interpretability and continuous learning. Experimental results show that the CEGNN is efficient and competitive in image classification, achieving an accuracyof 78.9% and a precision of 79.0% in a 10-class problem. This opens research avenues in certain applications, such as those dealing with medical images, satellite images, and image-based closed-loop autonomous systems.
The expanding cyber threat landscape has compelled organizations to adopt AI-driven security systems for robust defense against sophisticated attacks. This chapter explores artificial intelligence in cybersecurity, emphasizing its role in intelligent threat detection, analysis, and response. AI models, including supervised and unsupervised learning, deep learning, and reinforcement learning, have redefined cybersecurity by enabling behavior-based anomaly detection and automated threat mitigation. Key discussions highlight autonomous systems making real-time decisions, leveraging adaptive control loops, and employing self-healing mechanisms for resilience. This chapter also examines challenges in operational scalability, ethical implications of automation, and the necessity of human oversight in decision-making. The findings underscore the need for synergy between automation and human expertise to foster an intelligent, adaptive cyber defense ecosystem.
With an increased number of applications of machine learning models to support decision-making in critical domains, there is a pressing need to understand the internal behavior of these models. Essentially, explaining learning models to humans has expedited the development of methods to extract information and access models' inner components. Researchers have proposed approaches to compute explanations for different types of models. While these approaches are often effective in explaining complex relations and rough input-output maps, studies on model optimality, which trade off accuracy and interpretability are scarce. We conducted a study to understand the relationship between accuracy and interpretability of Hoeffding trees, developed from evolving data streams. We employed formal reasoning techniques, founded on theoretical guarantees, to generate subset-minimal explanations for a set of examples. Rankings of features, according to their importance to the model outcomes, were obtained. After computing model accuracy and interpretability, the least important feature based on the ranking was removed. By repeating this procedure, we leveraged the setup that leads to an optimal (accurate and interpretable) tree model. Application examples considered medical datasets, namely the Parkinson's Disease telemonitoring and the EEG eye-state datasets, to generate Hoeffding regression and classification trees, respectively. The study has shown that as tree interpretability increases, accuracy tends to decrease; however, an optimal solution can be established by balancing conflicting aspects.
We present an approach for data-driven modeling and evolving control of unknown dynamic systems called State-Space Evolving Granular Control. The approach is based on elements of granular computing, discrete state-space systems, and online learning. First, the structure and parameters of a granular model is developed from a stream of state data. The model is formed by information granules comprising first-order difference equations. Partial activation of granules gives global nonlinear approximation capability. The model is supplied with an algorithm that constantly updates the granules toward covering new data; however, keeping memory of previous patterns. A granular controller is derived from the granular model for parallel distributed compensation. Instead of difference equations, the content of a control granule is a gain matrix, which can be redesigned in real-time from the solution of a relaxed locally-valid linear matrix inequality derived from a Lyapunov function and bounded control-input conditions. We have shown asymptotic stabilization of a chaotic map assuming no previous knowledge about the source that produces the stream of data.
Artificial intelligence (AI) is currently being utilized in a wide range of sophisticated applications, but the outcomes of many AI models are challenging to comprehend and trust due to their black-box nature. Usually, it is essential to understand the reasoning behind an AI model’s decision-making. Thus, the need for eXplainable AI (XAI) methods for improving trust in AI models has arisen. XAI has become a popular research subject within the AI field in recent years. Existing survey papers have tackled the concepts of XAI, its general terms, and post-hoc explainability methods but there have not been any reviews that have looked at the assessment methods, available tools, XAI datasets, and other related aspects. Therefore, in this comprehensive study, we provide readers with an overview of the current research and trends in this rapidly emerging area with a case study example. The study starts by explaining the background of XAI, common definitions, and summarizing recently proposed techniques in XAI for supervised machine learning. The review divides XAI techniques into four axes using a hierarchical categorization system: (i) data explainability, (ii) model explainability, (iii) post-hoc explainability, and (iv) assessment of explanations. We also introduce available evaluation metrics as well as open-source packages and datasets with future research directions. Then, the significance of explainability in terms of legal demands, user viewpoints, and application orientation is outlined, termed as XAI concerns. This paper advocates for tailoring explanation content to specific user types. An examination of XAI techniques and evaluation was conducted by looking at 410 critical articles, published between January 2016 and October 2022, in reputed journals and using a wide range of research databases as a source of information. The article is aimed at XAI researchers who are interested in making their AI models more trustworthy, as well as towards researchers from other disciplines who are looking for effective XAI methods to complete tasks with confidence while communicating meaning from data.
This paper presents a method called Interval Incremental Learning (IIL) to capture spatial and temporal patterns in uncertain data streams. The patterns are represented by information granules and a granular rule base with the purpose of developing explainable human-centered computational models of virtual and physical systems. Fundamentally, interval data are either included into wider and more meaningful information granules recursively, or used for structural adaptation of the rule base. An Uncertainty-Weighted Recursive-Least-Squares (UW-RLS) method is proposed to update affine local functions associated with the rules. Online recursive procedures that build interval-based models from scratch and guarantee balanced information granularity are described. The procedures assure stable and understandable rule-based modeling. In general, the model can play the role of a predictor, a controller, or a classifier, with online sample-per-sample structural adaptation and parameter estimation done concurrently. The IIL method is aligned with issues and needs of the Internet of Things, Big Data processing, and eXplainable Artificial Intelligence. An application example concerning real-time land-vehicle localization and tracking in an uncertain environment illustrates the usefulness of the method. We also provide the Driving Through Manhattan interval dataset to foster future investigation.
As one of the three pillars in computational intelligence, fuzzy systems are a powerful mathematical tool widely used for modelling nonlinear problems with uncertainties. Fuzzy systems take the form of linguistic IF-THEN fuzzy rules that are easy to understand for human. In this sense, fuzzy inference mechanisms have been developed to mimic human reasoning and decision-making. From a data analytic perspective, fuzzy systems provide an effective solution to build precise predictive models from imprecise data with great transparency and interpretability, thus facilitating a wide range of real-world applications. This paper presents a systematic review of modern methods for autonomously learning fuzzy systems from data, with an emphasis on the structure and parameter learning schemes of mainstream evolving, evolutionary, reinforcement learning-based fuzzy systems. The main purpose of this paper is to introduce the underlying concepts, underpinning methodologies, as well as outstanding performances of the state-of-the-art methods. It serves as a one-stop guide for readers learning the representative methodologies and foundations of fuzzy systems or who desire to apply fuzzy-based autonomous learning in other scientific disciplines and applied fields.
Incrementally learning new information from a non-stationary stream of data, referred to as ‘continual learning’, is a key feature of natural intelligence, but a challenging problem for deep neural networks. In recent years, numerous deep learning methods for continual learning have been proposed, but comparing their performances is difficult due to the lack of a common framework. To help address this, we describe three fundamental types, or ‘scenarios’, of continual learning: task-incremental, domain-incremental and class-incremental learning. Each of these scenarios has its own set of challenges. To illustrate this, we provide a comprehensive empirical comparison of currently used continual learning strategies, by performing the Split MNIST and Split CIFAR-100 protocols according to each scenario. We demonstrate substantial differences between the three scenarios in terms of difficulty and in terms of the effectiveness of different strategies. The proposed categorization aims to structure the continual learning field, by forming a key foundation for clearly defining benchmark problems.
Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1) Understandability and G2 Clinical relevance. For the chosen explanation form, its specific XAI technique should be optimized for G3 Truthfulness, G4 Informative plausibility, and G5 Computational efficiency. Following the guidelines, we conducted a systematic evaluation on a novel problem of multi-modal medical image explanation with two clinical tasks, and proposed new evaluation metrics accordingly. Sixteen commonly-used heatmap XAI techniques were evaluated and found to be insufficient for clinical use due to their failure in G3 and G4. Our evaluation demonstrated the use of Clinical XAI Guidelines to support the design and evaluation of clinically viable XAI.
Interpretable machine learning aims at unveiling the reasons behind predictions returned by uninterpretable classifiers. One of the most valuable types of explanation consists of counterfactuals. A counterfactual explanation reveals what should have been different in an instance to observe a diverse outcome. For instance, a bank customer asks for a loan that is rejected. The counterfactual explanation consists of what should have been different for the customer in order to have the loan accepted. Recently, there has been an explosion of proposals for counterfactual explainers. The aim of this work is to survey the most recent explainers returning counterfactual explanations. We categorize explainers based on the approach adopted to return the counterfactuals, and we label them according to characteristics of the method and properties of the counterfactuals returned. In addition, we visually compare the explanations, and we report quantitative benchmarking assessing minimality, actionability, stability, diversity, discriminative power, and running time. The results make evident that the current state of the art does not provide a counterfactual explainer able to guarantee all these properties simultaneously.
Artificial intelligence (AI) provides many opportunities to improve private and public life. Discovering patterns and structures in large troves of data in an automated manner is a core component of data science, and currently drives applications in diverse areas such as computational biology, law and finance. However, such a highly positive impact is coupled with a significant challenge: how do we understand the decisions suggested by these systems in order that we can trust them? In this report, we focus specifically on data-driven methods—machine learning (ML) and pattern recognition models in particular—so as to survey and distill the results and observations from the literature. The purpose of this report can be especially appreciated by noting that ML models are increasingly deployed in a wide range of businesses. However, with the increasing prevalence and complexity of methods, business stakeholders in the very least have a growing number of concerns about the drawbacks of models, data-specific biases, and so on. Analogously, data science practitioners are often not aware about approaches emerging from the academic literature or may struggle to appreciate the differences between different methods, so end up using industry standards such as SHAP. Here, we have undertaken a survey to help industry practitioners (but also data scientists more broadly) understand the field of explainable machine learning better and apply the right tools. Our latter sections build a narrative around a putative data scientist, and discuss how she might go about explaining her models by asking the right questions. From an organization viewpoint, after motivating the area broadly, we discuss the main developments, including the principles that allow us to study transparent models vs. opaque models, as well as model-specific or model-agnostic post-hoc explainability approaches. We also briefly reflect on deep learning models, and conclude with a discussion about future research directions.
Electroencephalographic (EEG) recordings can be of great help in decoding the open/close hand’s motion preparation. To this end, cortical EEG source signals in the motor cortex (evaluated in the 1-s window preceding movement onset) are extracted by solving inverse problem through beamforming. EEG sources epochs are used as source-time maps input to a custom deep convolutional neural network (CNN) that is trained to perform 2-ways classification tasks: pre-hand close (HC) versus resting state (RE) and pre-hand open (HO) versus RE. The developed deep CNN works well (accuracy rates up to 89.65±5.29% for HC versus RE and 90.50±5.35% for HO versus RE), but the core of the present study was to explore the interpretability of the deep CNN to provide further insights into the activation mechanism of cortical sources during the preparation of hands’ sub-movements. Specifically, occlusion sensitivity analysis was carried out to investigate which cortical areas are more relevant in the classification procedure. Experimental results show a recurrent trend of spatial cortical activation across subjects. In particular, the central region (close to the longitudinal fissure) and the right temporal zone of the premotor together with the primary motor cortex appear to be primarily involved. Such findings encourage an in-depth study of cortical areas that seem to play a key role in hand’s open/close preparation.
Background
Explainability is one of the most heavily debated topics when it comes to the application of artificial intelligence (AI) in healthcare. Even though AI-driven systems have been shown to outperform humans in certain analytical tasks, the lack of explainability continues to spark criticism. Yet, explainability is not a purely technological issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration. This paper provides a comprehensive assessment of the role of explainability in medical AI and makes an ethical evaluation of what explainability means for the adoption of AI-driven tools into clinical practice.
Methods
Taking AI-based clinical decision support systems as a case in point, we adopted a multidisciplinary approach to analyze the relevance of explainability for medical AI from the technological, legal, medical, and patient perspectives. Drawing on the findings of this conceptual analysis, we then conducted an ethical assessment using the “Principles of Biomedical Ethics” by Beauchamp and Childress (autonomy, beneficence, nonmaleficence, and justice) as an analytical framework to determine the need for explainability in medical AI.
Results
Each of the domains highlights a different set of core considerations and values that are relevant for understanding the role of explainability in clinical practice. From the technological point of view, explainability has to be considered both in terms how it can be achieved and what is beneficial from a development perspective. When looking at the legal perspective we identified informed consent, certification and approval as medical devices, and liability as core touchpoints for explainability. Both the medical and patient perspectives emphasize the importance of considering the interplay between human actors and medical AI. We conclude that omitting explainability in clinical decision support systems poses a threat to core ethical values in medicine and may have detrimental consequences for individual and public health.
Conclusions
To ensure that medical AI lives up to its promises, there is a need to sensitize developers, healthcare professionals, and legislators to the challenges and limitations of opaque algorithms in medical AI and to foster multidisciplinary collaboration moving forward.
Mining and analysing streaming data is crucial for many applications, and this area of research has gained extensive attention over the past decade. However, there are several inherent problems that continue to challenge the hardware and the state-of-the art algorithmic solutions. Examples of such problems include the unbound size, varying speed and unknown data characteristics of arriving instances from a data stream. The aim of this research is to portray key challenges faced by algorithmic solutions for stream mining, particularly focusing on the prevalent issue of concept drift. A comprehensive discussion of concept drift and its inherent data challenges in the context of stream mining is presented, as is a critical, in-depth review of relevant literature. Current issues with the evaluative procedure for concept drift detectors is also explored, highlighting problems such as a lack of established base datasets and the impact of temporal dependence on concept drift detection. By exposing gaps in the current literature, this study suggests recommendations for future research which should aid in the progression of stream mining and concept drift detection algorithms.
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Supervised machine-learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? Models should be not only good, but also interpretable, yet the task of interpretation appears underspecified. The academic literature has provided diverse and sometimes non-overlapping motivations for interpretability and has offered myriad techniques for rendering interpretable models. Despite this ambiguity, many authors proclaim their models to be interpretable axiomatically, absent further argument. Problematically, it is not clear what common properties unite these techniques. This article seeks to refine the discourse on interpretability. First it examines the objectives of previous papers addressing interpretability, finding them to be diverse and occasionally discordant. Then, it explores model properties and techniques thought to confer interpretability, identifying transparency to humans and post hoc explanations as competing concepts. Throughout, the feasibility and desirability of different notions of interpretability are discussed. The article questions the oft-made assertions that linear models are interpretable and that deep neural networks are not. © 2018 Association for Computing Machinery. All rights reserved.
In the last years many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness sometimes at the cost of scarifying accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, delineating explicitly or implicitly its own definition of interpretability and explanation. The aim of this paper is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Information granules, along with their processing constitute a core of granular computing, which provides a unified conceptual and algorithmic framework for designing and analyzing intelligent systems. In this study, we engage a principle of justifiable granularity as a way of forming type-1 and type-2 information granules—granular interval-valued information granules, whose descriptors are intervals themselves rather than numeric entities. A two-phase design process is presented: first, intervals (viz. information granules of type-1) are constructed based on available experimental data. Second, considering the data that have not been “covered” by the intervals (the data one can refer to as residual granular data), we construct their bounds in the form of information granules (instead of numeric values) thereby giving rise to the concept of granular intervals, namely information granules of type-2. A series of experiments are provided that focus on sensor fusion formed with the aid of information granules and granular system modeling of type-1 and type-2.
Based on the notion of granular structures, a triarchic theory of granular computing integrates three important perspectives, namely, philosophy of structured thinking, methodology of structured problem solving, and mechanism of structured information processing. A granular structure comprises multiple levels that are populated by granules of varying granularity. While a single granular structure provides a multilevel understanding and representation of the reality or a problem, many granular structures give rise to multiview. With an emphasis on multilevel and multiview, the triarchic theory investigates systematically ideas and structured ways to thinking, problem solving, and information processing. It aims at a unified theory that synthesizes various approaches to granular computing.
Understanding the rationale behind the predictions produced by machine learning models is a necessary prerequisite for human to build confidence and trust for the intelligent systems. To tackle the problem of interpretability faced by black-box models, a fuzzy local surrogate model is proposed in this study to articulate the rationale for predictions to enhance the interpretability of the results of machine learning models. Fuzzy rule-based model comes with high interpretability since it is composed of a collection of readable rules, and thus is suitable for prediction interpretation. The general scheme of fuzzy local surrogate model is composed of the following phases: i) select data points around the instance of interest, for which we wish to explain the prediction result produced by the predictive model; ii) generate predictions for these newly selected data and weight the selected data based on the distance from the instance of interest; iii) a fuzzy rule-based model composed a collection of interpretable is constructed to approximate the weighted data and offer meaningful interpretation to the prediction result of the given instance. The proposed fuzzy model for explaining predictions is model-agnostic and could provide high estimation accuracy. The proposed methodology offers a significant original contribution to the interpretation of machine learning models. Experimental studies demonstrate the usefulness of the proposed fuzzy local surrogate model in providing local explanations.
As our dependence on intelligent machines continues to grow, so does the demand for more transparent and interpretable models. In addition, the ability to explain the model generally is now the gold standard for building trust and deployment of Artificial Intelligence (AI) systems in critical domains. Explainable Artificial Intelligence (XAI) aims to provide a suite of machine learning (ML) techniques that enable human users to understand, appropriately trust, and produce more explainable models. Selecting an appropriate approach for building an XAI-enabled application requires a clear understanding of the core ideas within XAI and the associated programming frameworks. We survey state-of-the-art programming techniques for XAI and present the different phases of XAI in a typical ML development process. We classify the various XAI approaches and using this taxonomy, discuss the key differences among the existing XAI techniques. Furthermore, concrete examples are used to describe these techniques that are mapped to programming frameworks and software toolkits. It is the intention that this survey will help stakeholders in selecting the appropriate approaches, programming frameworks, and software toolkits by comparing them through the lens of the presented taxonomy.
We introduce a novel model-agnostic system that explains the behavior of complex models with high-precision rules called anchors, representing local, "sufficient" conditions for predictions. We propose an algorithm to efficiently compute these explanations for any black-box model with high-probability guarantees. We demonstrate the flexibility of anchors by explaining a myriad of different models for different domains and tasks. In a user study, we show that anchors enable users to predict how a model would behave on unseen instances with less effort and higher precision, as compared to existing linear explanations or no explanations.
Emotion recognition has become a need for more realistic and interactive machines and computer systems. The greatest challenge is the availability of high-performance algorithms to effectively manage individual differences and nonstationarities in physiological data, i.e., algorithms that customize models to users with no subject-specific calibration data. We describe an evolving Gaussian Fuzzy Classifier (eGFC), which is supported by a semi-supervised learning algorithm to recognize emotion patterns from electroencephalogram (EEG) data streams. We extract features from the Fourier spectrum of EEG data. The data are provided by 28 individuals playing the games ‘Train Sim World’, ‘Unravel’, ‘Slender The Arrival’, and ‘Goat Simulator’ – a public dataset. Different emotions prevail, namely, boredom, calmness, horror and joy. We analyze individual electrodes, time window lengths, and frequency bands on the accuracy of user-independent eGFCs. We conclude that both brain hemispheres may assist classification, especially electrodes on the frontal (Af3-Af4), occipital (O1-O2), and temporal (T7-T8) areas. We observe that patterns may be eventually found in any frequency band; however, the Alpha (8-13Hz), Delta (1-4Hz), and Theta (4-8Hz) bands, in this order, are more correlated with the emotion classes. eGFC has shown to be effective for real-time learning of EEG data. It reaches a 72.2% accuracy using a variable rule base, 10-second windows, and 1.8ms/sample processing time in a highly-stochastic time-varying 4-class classification problem.
In this study, based on the principle of justifiable granularity, a method for designing multi-dimensional information granules is proposed. To design information granules, the correlations among the different variables within the data and their confidence levels are considered. The designed information granules reveal the relationships present in the experimental data and help to capture more features of the original data. In addition, a strategy for the exclusion of inhibitory data is considered, making the design of information granules more focused. Several experimental studies are conducted to quantify the effectiveness of the proposed method.
We introduce an incremental learning method for the optimal construction of rule-based granular systems from numerical data streams. The method is developed within a multiobjective optimization framework considering the specificity of information, model compactness, and variability and granular coverage of the data. We use α-level sets over Gaussian membership functions to set model granularity and operate with hyperrectangular forms of granules in nonstationary environments. The resulting rule-based systems are formed in a formal and systematic fashion. They can be useful in time series modeling, dynamic system identification, predictive analytics, and adaptive control. Precise estimates and enclosures are given by linear piecewise and inclusion functions related to optimal granular mappings.
Time-varying classifiers, namely, evolving classifiers, play an important role in a scenario in which information is available as a never-ending online data stream. We present a new unsupervised learning method for numerical data called evolving Internal-eXternal Fuzzy clustering method (Fuzzy eIX). We develop the notion of double-boundary fuzzy granules and elaborate on its implications. Type 1 and type 2 fuzzy inference systems can be obtained from the projection of Fuzzy eIX granules. We perform the principle of the balanced information granularity within Fuzzy eIX classifiers to achieve a higher level of model understandability. Internal and external granules are updated from a numerical data stream at the same time that the global granular structure of the classifier is autonomously evolved. A synthetic nonstationary problem called Rotation of Twin Gaussians shows the behavior of the classifier. The Fuzzy eIX classifier could keep up with its accuracy in a scenario in which offline-trained classifiers would clearly have their accuracy drastically dropped.
Missing values are common in real-world data stream applications. This paper proposes a modified evolving granular fuzzy rule-based model for function approximation and time series prediction in an online context where values may be missing. The fuzzy model is equipped with an incremental learning algorithm that simultaneously imputes missing data and adapts model parameters and structure over time. The evolving Fuzzy Granular Predictor (eFGP) handles single and multiple missing values on data samples by developing reduced-term consequent polynomials and utilizing time-varying granules. Missing at random (MAR) and missing completely at random (MCAR) values in nonstationary data streams are approached. Experiments to predict monthly weather conditions, the number of bikes hired on a daily basis, and the sound pressure on an airfoil from incomplete data streams show the usefulness of eFGP models. Results were compared with those of state-of-the-art fuzzy and neuro-fuzzy evolving modeling methods. A statistical hypothesis test shows that eFGP outperforms other evolving intelligent methods in online MAR and MCAR settings, regardless of the application.
(https://ieeexplore.ieee.org/document/8801860)
Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society. The way forward is to design models that are inherently interpretable. This Perspective clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare and computer vision. There has been a recent rise of interest in developing methods for ‘explainable AI’, where models are created to explain how a first ‘black box’ machine learning model arrives at a specific decision. It can be argued that instead efforts should be directed at building inherently interpretable models in the first place, in particular where they are applied in applications that directly affect human lives, such as in healthcare and criminal justice.
Interpretability is a fundamental property for the acceptance of machine learning models in highly regulated areas. Recently, deep neural networks gained the attention of the scientific community due to their high accuracy in vast classification problems. However, they are still seen as black-box models where it is hard to understand the reasons for the labels that they generate. This paper proposes a deep model with monotonic constraints that generates complementary explanations for its decisions both in terms of style and depth. Furthermore, an objective framework for the evaluation of the explanations is presented. Our method is tested on two biomedical datasets and demonstrates an improvement in relation to traditional models in terms of quality of the explanations generated.
When a Deep Neural Network makes a misprediction, it can be challenging for a developer to understand why. While there are many models for interpretability in terms of predictive features, it may be more natural to isolate a small set of training examples that have the greatest influence on the prediction. However, it is often the case that every training example contributes to a prediction in some way but with varying degrees of responsibility. We present Partition Aware Local Model (PALM), which is a tool that learns and summarizes this responsibility structure to aide machine learning debugging. PALM approximates a complex model (e.g., a deep neural network) using a two-part surrogate model: a meta-model that partitions the training data, and a set of sub-models that approximate the patterns within each partition. These sub-models can be arbitrarily complex to capture intricate local patterns. However, the meta-model is constrained to be a decision tree. This way the user can examine the structure of the meta-model, determine whether the rules match intuition, and link problematic test examples to responsible training data efficiently. Queries to PALM are nearly 30x faster than nearest neighbor queries for identifying relevant data, which is a key property for interactive applications.
Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one.
In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally varound the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.
The study introduces and discusses a principle of justifiable granularity, which supports a coherent way of designing information granules in presence of experimental evidence (either of numerical or granular character). The term “justifiable” pertains to the construction of the information granule, which is formed in such a way that it is (a) highly legitimate (justified) in light of the experimental evidence, and (b) specific enough meaning it comes with a well-articulated semantics (meaning). The design process associates with a well-defined optimization problem with the two requirements of experimental justification and specificity. A series of experiments is provided as well as a number of constructs carried for various formalisms of information granules (intervals, fuzzy sets, rough sets, and shadowed sets) are discussed as well.
This paper introduces a granular neural network framework for evolving fuzzy system modeling from fuzzy data streams. The evolving granular neural network (eGNN) is able to handle gradual and abrupt parameter changes typical of nonstationary (online) environments. eGNN builds interpretable multi-sized local models using fuzzy neurons for information fusion. An online incremental learning algorithm develops the neural network structure from the information contained in data streams. We focus on trapezoidal fuzzy intervals and objects with trapezoidal membership function representation. More precisely, the framework considers triangular, interval, and numeric types of data to construct granular fuzzy models as particular arrangements of trapezoids. Application examples in classification and function approximation in material and biomedical engineering are used to evaluate and illustrate the neural network usefulness. Simulation results suggest that the eGNN fuzzy modeling approach can handle fuzzy data successfully and outperforms alternative state-of-the-art approaches in terms of accuracy, transparency and compactness.
Uncertainty is an attribute of information. The path-breaking work of Shannon has led to a universal acceptance of the thesis that information is statistical in nature. Concomitantly, existing theories of uncertainty are based on probability theory. The generalized theory of uncertainty (GTU) departs from existing theories in essential ways. First, the thesis that information is statistical in nature is replaced by a much more general thesis that information is a generalized constraint, with statistical uncertainty being a special, albeit important case. Equating information to a generalized constraint is the fundamental thesis of GTU. Second, bivalence is abandoned throughout GTU, and the foundation of GTU is shifted from bivalent logic to fuzzy logic. As a consequence, in GTU everything is or is allowed to be a matter of degree or, equivalently, fuzzy. Concomitantly, all variables are, or are allowed to be granular, with a granule being a clump of values drawn together by a generalized constraint. And third, one of the principal objectives of GTU is achievement of NL-capability, that is, the capability to operate on information described in natural language. NL-capability has high importance because much of human knowledge, including knowledge about probabilities, is described in natural language. NL-capability is the focus of attention in the present paper. The centerpiece of GTU is the concept of a generalized constraint. The concept of a generalized constraint is motivated by the fact that most real-world constraints are elastic rather than rigid, and have a complex structure even when simple in appearance. The paper concludes with examples of computation with uncertain information described in natural language.
Interpretability is acknowledged as the main advantage of fuzzy systems and it should be given a main role in fuzzy modeling. Classical systems are viewed as black boxes because mathematical formulas set the mapping between inputs and outputs. On the contrary, fuzzy systems (if they are built regarding some constraints) can be seen as gray boxes in the sense that every element of the whole system can be checked and understood by a human being. Interpretability is essential for those applications with high human interaction, for instance decision support systems in fields like medicine, economics, etc. Since interpretability is not guaranteed by definition, a huge effort has been done to find out the basic constraints to be superimposed during the fuzzy modeling process. People talk a lot about interpretability but the real meaning is not clear. Understanding of fuzzy systems is a subjective task which strongly depends on the background (experience, preferences, and knowledge) of the person who makes the assessment. As a consequence, although there have been a few attempts to define interpretability indices, there is still not a universal index widely accepted. As part of this work, with the aim of evaluating the most used indices, an experimental analysis (in the form of a web poll) was carried out yielding some useful clues to keep in mind regarding interpretability assessment. Results extracted from the poll show the inherent subjectivity of the measure because we collected a huge diversity of answers completely different at first glance. However, it was possible to find out some interesting user profiles after comparing carefully all the answers. It can be concluded that defining a numerical index is not enough to get a widely accepted index. Moreover, it is necessary to define a fuzzy index easily adaptable to the context of each problem as well as to the user quality criteria.
We introduce the concept of specificity and indicate its importance as a measure of uncertainty for information represented using fuzzy sets or possibility distributions. We provide formal measures of specificity for variables whose domain is an interval of the real line. Similarity relations are discussed and formalized. The class of width-based similarity relations is introduced. We extend the measure of specificity to allow for the effect of an underlying similarity relationship. The manifestation of this extension for a number of different similarity relations is investigated. Particularly notable results are obtained for width-based similarity relationships. More generally we note the connection between the size of a granule of information and its uncertainty. Motivated by this we investigate the effect of a similarity relationship on the perception of distance in the underlying space and use this to develop an extension of the specificity measure.
The judicial demand for explainable artificial intelligence
- A Deeks
Local rule-based explanations of black box decision systems
- Guidotti