No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
GridEx: An Algorithm for Knowledge Extraction from Black-Box Regressors
Abstract
Knowledge-extraction methods are applied to ML-based predictors to attain explainable representations of their operation when the lack of interpretable results constitutes a problem. Several algorithms have been proposed for knowledge extraction, mostly focusing on the extraction of either lists or trees of rules. Yet, most of them only support supervised learning – and, in particular, classification – tasks. Iter is among the few rule-extraction methods capable of extracting symbolic rules out of sub-symbolic regressors. However, its performance – here intended as the interpretability of the rules it extracts – easily degrades as the complexity of the regression task at hand increases.
ResearchGate has not been able to resolve any citations for this publication.
Recently, the eXplainable AI (XAI) research community has focused on developing methods making Machine Learning (ML) predictors more interpretable and explainable. Unfortunately, researchers are struggling to converge towards an unambiguous definition of notions such as interpretation, or, explanation—which are often (and mistakenly) used interchangeably. Furthermore, despite the sound metaphors that Multi-Agent System (MAS) could easily provide to address such a challenge, and agent-oriented perspective on the topic is still missing. Thus, this paper proposes an abstract and formal framework for XAI-based MAS, reconciling notions, and results from the literature.
Deep neural networks are vulnerable to adversarial attacks and hard to interpret because of their black-box nature. The recently proposed invertible network is able to accurately reconstruct the inputs to a layer from its outputs, thus has the potential to unravel the black-box model. An invertible network classifier can be viewed as a two-stage model: (1) invertible transformation from input space to the feature space; (2) a linear classifier in the feature space. We can determine the decision boundary of a linear classifier in the feature space; since the transform is invertible, we can invert the decision boundary from the feature space to the input space. Furthermore, we propose to determine the projection of a data point onto the decision boundary, and define explanation as the difference between data and its projection. Finally, we propose to locally approximate a neural network with its first-order Taylor expansion, and define feature importance using a local linear model. We provide the implementation of our method: https://github.com/juntang-zhuang/explain_invertible.
We propose an abstract framework for XAI based on MAS encompassing the main definitions and results from the literature, focussing on the key notions of interpretation and explanation. KEYWORDS XAI; Multi-Agent Systems; Abstract Framework ACM Reference Format:
Understanding how a learned black box works is of crucial interest for the future of Machine Learning. In this paper, we pioneer the question of the global interpretability of learned black box models that assign numerical values to symbolic sequential data. To tackle that task, we propose a spectral algorithm for the extraction of weighted automata (WA) from such black boxes. This algorithm does not require the access to a dataset or to the inner representation of the black box: the inferred model can be obtained solely by querying the black box, feeding it inputs and analyzing its outputs. Experiments using Recurrent Neural Networks (RNN) trained on a wide collection of synthetic datasets and 2 real datasets show that the obtained approximation is of great quality.
Supervised machine-learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? Models should be not only good, but also interpretable, yet the task of interpretation appears underspecified. The academic literature has provided diverse and sometimes non-overlapping motivations for interpretability and has offered myriad techniques for rendering interpretable models. Despite this ambiguity, many authors proclaim their models to be interpretable axiomatically, absent further argument. Problematically, it is not clear what common properties unite these techniques. This article seeks to refine the discourse on interpretability. First it examines the objectives of previous papers addressing interpretability, finding them to be diverse and occasionally discordant. Then, it explores model properties and techniques thought to confer interpretability, identifying transparency to humans and post hoc explanations as competing concepts. Throughout, the feasibility and desirability of different notions of interpretability are discussed. The article questions the oft-made assertions that linear models are interpretable and that deep neural networks are not. © 2018 Association for Computing Machinery. All rights reserved.
In the last years many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness sometimes at the cost of scarifying accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, delineating explicitly or implicitly its own definition of interpretability and explanation. The aim of this paper is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Making diagnosis by learning from examples is a typical field of artificial neural networks. However, justifications of network responses are difficult to obtain, especially when input examples have analog variables. We propose a particular multi-layer Perceptron model in which explanations of responses are obtained through symbolic rules. The originality of this model consists in its architecture. Experiments using three datasets related to breast cancer diagnosis, coronary heart disease and thyroid dysfunctions have shown high mean predictive accuracy (respectively: 96.3%, 90.0%, 99.3%). Comparisons with the C4.5 algorithm, which builds inductive decision trees, have shown that the predictive accuracy of both approaches is roughly the same, with neural networks slightly more accurate.
With several good research groups actively working in machine learning (ML) ap-proaches, we have now the concept of self-containing machine learning solutions that oftentimes work out-of-the-box leading to the concept of ML black-boxes. Although it is important to have such black-boxes helping researchers to deal with several problems nowadays, it comes with an inherent problem increasingly more evident: we have ob-served that researchers and students are progressively relying on ML black-boxes and, usually, achieving results without knowing the machinery of the classifiers. In this regard, this paper discusses the use of machine learning black-boxes and poses the question of how far we can get using these out-of-the-box solutions instead of going deeper into the machinery of the classifiers. The paper focuses on three aspects of classifiers: (1) the way they compare examples in the feature space; (2) the impact of using features with variable dimensionality; and (3) the impact of using binary classifiers to solve a multi-class problem. We show how knowledge about the classifier's machinery can improve the results way beyond out-of-the-box machine learning solutions.
Credit-risk evaluation is a very important management science problem in the financial analysis area. Neural Networks have received a lot of attention because of their universal approximation property. They have a high prediction accuracy rate, but it is not easy to understand how they reach their decisions. In this paper, we present a real-life credit-risk data set and analyze it by using the NeuroRule extraction technique and the WEKA software. The results were considered very satisfactory.
The inherent black-box nature of neural networks is an im-portant drawback with respect to the problem of knowledge discovery from databases. In this work our aim is to extract rules from multi-layer perceptrons. This is a starting point to explain the basis of neural net-work solutions for knowledge discovery to experts of domain applications. Our approach consists in characterizing discriminant hyper-plane fron-tiers built by a special neural network model denoted to as Discretized Interpretable Multi Layer Perceptron (DIMLP). Rules are extracted in polynomial time with respect to the size of the problem and the size of the network. Further, the degree of matching between extracted rules and neural network responses is 100%. We apply DIMLP to 9 databases related to the medical diagnosis domain in which for some of them it gives better average predictive accuracy than standard multi-layer per-ceptrons and C4.5 decision trees. Finally, the quality of rules generated from DIMLP networks is compared to those related to decision trees.
It is becoming increasingly apparent that, without some form of explanation capability, the full potential of trained artificial neural networks (ANNs) may not be realised. This survey gives an overview of techniques developed to redress this situation. Specifically, the survey focuses on mechanisms, procedures, and algorithms designed to insert knowledge into ANNs (knowledge initialisation), extract rules from trained ANNs (rule extraction), and utilise ANNs to refine existing rule bases (rule refinement). The survey also introduces a new taxonomy for classifying the various techniques, discusses their modus operandi, and delineates criteria for evaluating their efficacy.
Breast cancer relapse prediction is an important step in the complex decision-making process of deciding the type of treatment to be applied to patients after surgery. Some non-linear models, like neural networks, have been successfully applied to this task but they suffer from the problem of extracting the underlying rules, and knowing how the methods operate can help to a better understanding of the cancer relapse problem. A recently introduced constructive algorithm (DASG) that creates compact neural network architectures is applied to a dataset of early breast cancer patients with the aim of testing the predictive ability of the new method. The DASG method works with Boolean input data and for that reason a transformation procedure was applied to the original data. The degradation in the predictive performance due to the transformation of the data is also analyzed using the new method and other standard algorithms.
Most accurate predictions are typically obtained by learning machines with
complex feature spaces (as e.g. induced by kernels). Unfortunately, such
decision rules are hardly accessible to humans and cannot easily be used to
gain insights about the application domain. Therefore, one often resorts to
linear models in combination with variable selection, thereby sacrificing some
predictive power for presumptive interpretability. Here, we introduce the
Feature Importance Ranking Measure (FIRM), which by retrospective analysis of
arbitrary learning machines allows to achieve both excellent predictive
performance and superior interpretation. In contrast to standard raw feature
weighting, FIRM takes the underlying correlation structure of the features into
account. Thereby, it is able to discover the most relevant features, even if
their appearance in the training data is entirely prevented by noise. The
desirable properties of FIRM are investigated analytically and illustrated in
simulations.
In life sciences, interpretability of machine learning models is as important as their prediction accuracy. Linear models are probably the most frequently used methods for assessing feature relevance, despite their relative inflexibility. However, in the past years effective estimators of feature relevance have been derived for highly complex or non-parametric models such as support vector machines and RandomForest (RF) models. Recently, it has been observed that RF models are biased in such a way that categorical variables with a large number of categories are preferred.
In this work, we introduce a heuristic for normalizing feature importance measures that can correct the feature importance bias. The method is based on repeated permutations of the outcome vector for estimating the distribution of measured importance for each variable in a non-informative setting. The P-value of the observed importance provides a corrected measure of feature importance. We apply our method to simulated data and demonstrate that (i) non-informative predictors do not receive significant P-values, (ii) informative variables can successfully be recovered among non-informative variables and (iii) P-values computed with permutation importance (PIMP) are very helpful for deciding the significance of variables, and therefore improve model interpretability. Furthermore, PIMP was used to correct RF-based importance measures for two real-world case studies. We propose an improved RF model that uses the significant variables with respect to the PIMP measure and show that its prediction accuracy is superior to that of other existing models.
R code for the method presented in this article is available at http://www.mpi-inf.mpg.de/ approximately altmann/download/PIMP.R CONTACT: altmann@mpi-inf.mpg.de, laura.tolosi@mpi-inf.mpg.de
Supplementary data are available at Bioinformatics online.
Support vector machines (SVMs) have shown superior performance compared to other machine learning techniques, especially in classification problems. Yet one limitation of SVMs is the lack of an explanation capability which is crucial in some applications, e.g. in the medical and security domains. In this paper, a novel approach for eclectic rule- extraction from support vector machines is presented. This approach utilizes the knowledge acquired by the SVM and represented in its support vectors as well as the parameters associated with them. The approach includes three stages; training, propositional rule- extraction and rule quality evaluation. Results from four different experiments have demonstrated the value of the approach for extracting comprehensible rules of high accuracy and fidelity.
The more intelligent systems based on sub-symbolic techniques pervade our everyday lives, the less human can understand them. This is why symbolic approaches are getting more and more attention in the general effort to make AI interpretable, explainable, and trustable. Understanding the current state of the art of AI techniques integrating symbolic and sub-symbolic approaches is then of paramount importance, nowadays-in particular in the XAI perspective. This is why this paper provides an overview of the main symbolic/sub-symbolic integration techniques, focussing in particular on those targeting explainable AI systems.
Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society. The way forward is to design models that are inherently interpretable. This Perspective clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare and computer vision. There has been a recent rise of interest in developing methods for ‘explainable AI’, where models are created to explain how a first ‘black box’ machine learning model arrives at a specific decision. It can be argued that instead efforts should be directed at building inherently interpretable models in the first place, in particular where they are applied in applications that directly affect human lives, such as in healthcare and criminal justice.
The vast majority of the literature evaluates the performance of classification models using only the criterion of predictive accuracy. This paper reviews the case for considering also the comprehensibility (interpretability) of classification models, and discusses the interpretability of five types of classification models, namely decision trees, classification rules, decision tables, nearest neighbors and Bayesian network classifiers. We discuss both interpretability issues which are specific to each of those model types and more generic interpretability issues, namely the drawbacks of using model size as the only criterion to evaluate the comprehensibility of a model, and the use of monotonicity constraints to improve the comprehensibility and acceptance of classification models by users.
Keyword extraction is vital for Knowledge Management System, Information Retrieval System, and Digital Libraries as well as for general browsing of the web. Keywords are often the basis of document processing methods such as clustering and retrieval since processing all the words in the document can be slow. Common models for automating the process of keyword extraction are usually done by using several statistics-based methods such as Bayesian, K-Nearest Neighbor, and Expectation-Maximization. These models are limited by word-related features that can be used since adding more features will make the models more complex and difficult to comprehend. In this research, a Neural Network, specifically a backpropagation network, will be used in generalizing the relationship of the title and the content of articles in the archive by following word features other than TF-IDF, such as position of word in the sentence, paragraph, or in the entire document, and formats such as heading, and other attributes defined beforehand. In order to explain how the backpropagation network works, a rule extraction method will be used to extract symbolic data from the resulting backpropagation network. The rules extracted can then be transformed into decision trees performing almost as accurate as the network plus the benefit of being in an easily comprehensible format.
Credit-risk evaluation is a very challenging and important management science problem in the domain of financial analysis. Many classification methods have been suggested in the literature to tackle this problem. Neural networks, especially, have received a lot of attention because of their universal approximation property. However, a major drawback associated with the use of neural networks for decision making is their lack of explanation capability. While they can achieve a high predictive accuracy rate, the reasoning behind how they reach their decisions is not readily available. In this paper, we present the results from analysing three real-life credit-risk data sets using neural network rule extraction techniques. Clarifying the neural network decisions by explanatory rules that capture the learned knowledge embedded in the networks can help the credit-risk manager in explaining why a particular applicant is classified as either bad or good. Furthermore, we also discuss how these rules can be visualized as a decision table in a compact and intuitive graphical format that facilitates easy consultation. It is concluded that neural network rule extraction and decision tables are powerful management tools that allow us to build advanced and userfriendly decision-support systems for credit-risk evaluation.
Various benchmarking studies have shown that artificial neural networks and support vector machines have a superior performance
when compared to more traditional machine learning techniques. The main resistance against these newer techniques is based
on their lack of interpretability: it is difficult for the human analyst to understand the motivation behind these models’
decisions. Various rule extraction techniques have been proposed to overcome this opacity restriction. However, most of these
extraction techniques are devised for classification and only few algorithms can deal with regression problems.
In this paper, we present ITER, a new algorithm for pedagogical regression rule extraction. Based on a trained ‘black box’
model, ITER is able to extract human-understandable regression rules. Experiments show that the extracted model performs well
in comparison with CART regression trees and various other techniques.
In recent years, support vector machines (SVMs) were successfully applied to a wide range of applications. However, since the classifier is described as a complex mathematical function, it is rather incomprehensible for humans. This opacity property prevents them from being used in many real-life applications where both accuracy and comprehensibility are required, such as medical diagnosis and credit risk evaluation. To overcome this limitation, rules can be extracted from the trained SVM that are interpretable by humans and keep as much of the accuracy of the SVM as possible. In this paper, we will provide an overview of the recently proposed rule extraction techniques for SVMs and introduce two others taken from the artificial neural networks domain, being Trepan and G-REX. The described techniques are compared using publicly available datasets, such as Ripley’s synthetic dataset and the multi-class iris dataset. We will also look at medical diagnosis and credit scoring where comprehensibility is a key requirement and even a regulatory recommendation. Our experiments show that the SVM rule extraction techniques lose only a small percentage in performance compared to SVMs and therefore rank at the top of comprehensible classification techniques.
Many systems have been developed for constructing decision trees from collections of examples. Although the decision trees generated by these methods are accurate and efficient, they often suffer the disadvantage of excessive complexity and are therefore incomprehensible to experts. It is questionable whether opaque structures of this kind can be described as knowledge, no matter how well they function. This paper discusses techniques for simplifying decision trees while retaining their accuracy. Four methods are described, illustrated, and compared on a test-bed of decision trees from a variety of domains.
While feedforward neural networks have been widely accepted as effective tools for solving classification problems, the issue of finding the best network architecture remains unresolved, particularly so in real-world problem settings. We address this issue in the context of credit card screening, where it is important to not only find a neural network with good predictive performance but also one that facilitates a clear explanation of how it produces its predictions. We show that minimal neural networks with as few as one hidden unit provide good predictive accuracy, while having the added advantage of making it easier to generate concise and comprehensible classification rules for the user. To further reduce model size, a novel approach is suggested in which network connections from the input units to this hidden unit are removed by a very straightaway pruning procedure. In terms of predictive accuracy, both the minimized neural networks and the rule sets generated from them are shown to compare favorably with other neural network based classifiers. The rules generated from the minimized neural networks are concise and thus easier to validate in a real-life setting.
This paper proposes a new framework and method for extracting regression rules from neural networks trained with multivariate data containing both nominal and numeric variables. Each regression rule is expressed as a pair of a logical formula on the conditional part over nominal variables and a polynomial equation on the action part over numeric variables. The proposed extraction method first generates one such regression rule for each training sample, then utilizes the kappa-means algorithm to generate a much smaller set of rules having more general conditions, where the number of distinct polynomial equations is determined through cross-validation. Finally, this method invokes decision-tree induction to form logical formulae of nominal conditions as conditional parts of final regression rules. Experiments using four data sets show that our method works well in extracting quite accurate and interesting regression rules.
In many neural network applications, the understanding of the network functionality is an important issue. Trained neural networks are often termed as "black boxes" which do not allow to get a deeper insight into the relationships between the input (feature) and output spaces. In the past years some researchers addressed the problem of rule extraction from trained neural networks. This article investigates the properties and results of different rule extraction techniques in a specific application area: The detection of intrusions in computer networks. This application area is chosen because intrusion detection and the development of appropriate intrusion detection systems (IDS) gains more and more importance with the rapidly increasing impact of the Internet.
Neural networks (NNs) have been successfully applied to solve a
variety of application problems including classification and function
approximation. They are especially useful as function approximators
because they do not require prior knowledge of the input data
distribution and they have been shown to be universal approximators. In
many applications, it is desirable to extract knowledge that can explain
how Me problems are solved by the networks. Most existing approaches
have focused on extracting symbolic rules for classification. Few
methods have been devised to extract rules from trained NNs for
regression. This article presents an approach for extracting rules from
trained NNs for regression. Each rule in the extracted rule set
corresponds to a subregion of the input space and a linear function
involving the relevant input attributes of the data approximates the
network output for all data samples in this subregion. Extensive
experimental results on 32 benchmark data sets demonstrate the
effectiveness of the proposed approach in generating accurate regression
rules
Although artificial neural networks can represent a variety of
complex systems with a high degree of accuracy, these connectionist
models are difficult to interpret. This significantly limits the
applicability of neural networks in practice, especially where a premium
is placed on the comprehensibility or reliability of systems. A novel
artificial neural-network decision tree algorithm (ANN-DT) is therefore
proposed, which extracts binary decision trees from a trained neural
network. The ANN-DT algorithm uses the neural network to generate
outputs for samples interpolated from the training data set. In contrast
to existing techniques, ANN-DT can extract rules from feedforward neural
networks with continuous outputs. These rules are extracted from the
neural network without making assumptions about the internal structure
of the neural network or the features of the data. A novel attribute
selection criterion based on a significance analysis of the variables on
the neural-network output is examined. It is shown to have significant
benefits in certain cases when compared with the standard criteria of
minimum weighted variance over the branches. In three case studies the
ANN-DT algorithm compared favorably with CART, a standard decision tree
algorithm
Neural networks, despite their empirically-proven abilities, have been little used for the refinement of existing knowledge because this task requires a three-step process. First, knowledge in some form must be inserted into a neural network. Second, the network must be refined. Third, knowledge must be extracted from the network. We have previously described a method for the first step of this process. Standard neural learning techniques can accomplish the second step. In this paper, we propose and empirically evaluate a method for the final, and possibly most difficult, step. This method efficiently extracts symbolic rules from trained neural networks. The four major results of empirical tests of this method are that the extracted rules: (1) closely reproduce (and can even exceed) the accuracy of the network from which they are extracted; (2) are superior to the rules produced by methods that directly refine symbolic rules; (3) are superior to those produced by previous techniques fo...
We discuss an approach to constructing composite features during the induction of decision trees. The composite features correspond to m-of-n concepts. There are three goals of this research. First, we explore a family of greedy methods for building m-ofn concepts (one of which, GS, is described in this paper). Second, we show how these concepts can be formed as internal nodes of decision trees, serving as a bias to the learner. Finally, we evaluate the method on several artificially generated and naturally occurring data sets to determine the effects of this bias. 1 INTRODUCTION In this paper, we discuss an approach to constructing composite features during the induction of decision trees (Quinlan, 1986). Pagallo and Haussler (1990) describe the FRINGE algorithm that analyzes a decision tree and creates new features that are conjunctions of existing features. After the new features are constructed, the induction algorithm is run again and may incorporate the conjunctive features in ...
Concepts learned by neural networks are difficult to understand because they are represented using large assemblages of real-valued parameters. One approach to understanding trained neural networks is to extract symbolic rules that describe their classification behavior. There are several existing rule-extraction approaches that operate by searching for such rules. We present a novel method that casts rule extraction not as a search problem, but instead as a learning problem. In addition to learning from training examples, our method exploits the property that networks can be efficiently queried. We describe algorithms for extracting both conjunctive and M-of-N rules, and present experiments that show that our method is more efficient than conventional search-based approaches. 1 INTRODUCTION A problem that arises when neural networks are used for supervised learning tasks is that, after training, it is usually difficult to understand the concept representations formed by the networks....
A significant limitation of neural networks is that the representations they learn are usually incomprehensible to humans. We present a novel algorithm, Trepan, for extracting comprehensible, symbolic representations from trained neural networks. Our algorithm uses queries to induce a decision tree that approximates the concept represented by a given network. Our experiments demonstrate that Trepan is able to produce decision trees that maintain a high level of fidelity to their respective networks while being comprehensible and accurate. Unlike previous work in this area, our algorithm is general in its applicability and scales well to large networks and problems with high-dimensional input spaces.
Neural networks have been widely used as tools for prediction in medicine. We expect to see even more applications of neural networks for medical diagnosis as recently developed neural network rule extraction algorithms make it possible for the decision process of a trained network to be expressed as classification rules. These rules are more comprehensible to a human user than the classification process of the networks which involves complex nonlinear mapping of the input data. This paper reports the results from two neural network rule extraction techniques, NeuroLinear and NeuroRule applied to the diagnosis of hepatobiliary disorders. The data set consists of nine measurements collected from patients in a Japanese hospital and these measurements have continuous values. NeuroLinear generates piece-wise linear discriminant functions for this data set. The continuous measurements have previously been discretized by domain experts. NeuroRule is applied to the discretized data...
Building creditrisk evaluation expert systems using neural network rule extraction and decision tables
- B Baesens
- R Setiono
- V De Lille
- S Viaene
- J Vanthienen
Baesens, B., Setiono, R., De Lille, V., Viaene, S., Vanthienen, J.: Building creditrisk evaluation expert systems using neural network rule extraction and decision
tables. In: ICIS 2001 Proceedings, vol. 20 (2001). http://aisel.aisnet.org/icis2001/
20
Interpretable narrative explanation for ML predictors with LP: a case study for XAI
- R Calegari
- G Ciatto
- J Dellaluce
- A Omicini
Calegari, R., Ciatto, G., Dellaluce, J., Omicini, A.: Interpretable narrative explanation for ML predictors with LP: a case study for XAI. In: Bergenti, F., Monica,
S. (eds.) WOA 2019-20th Workshop "From Objects to Agents", CEUR Workshop
Proceedings, vol. 2404, pp. 105-112. Sun SITE Central Europe, RWTH Aachen
University, 26-28 June 2019. http://ceur-ws.org/Vol-2404/paper16.pdf
- J R Quinlan
Quinlan, J.R.: C4.5: Programming for Machine Learning. Morgan Kauffmann 38,
48 (1993)