No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Interpretable machine learning: Fundamental principles and 10 grand challenges
... Supervised learning is commonly used in prediction and classification problems, where the objective is to predict a specific outcome or category, although numerical values can also be predicted through regression models. Decision trees, scoring systems, generalised additive models, and case-based reasoning are among the primary techniques used in various supervised learning algorithms [57]. Each algorithm has its own specific characteristics and uses. ...
... Logistic regression is mainly used for binary classification tasks, although it could also be useful for multi-class problems, by modelling the probability of an event occurring based on input features [31]. Recursive partitioning is a technique commonly used in decision trees, where the data are recursively split into subsets based on certain conditions of features [57]. Random forest is an ensemble learning method that combines multiple decision trees to improve prediction accuracy and reduce overfitting [17,31]. ...
... Random forest is an ensemble learning method that combines multiple decision trees to improve prediction accuracy and reduce overfitting [17,31]. The k-nearest neighbour method classifies or predicts the value of a data point based on the values of its k-nearest neighbours in the feature space [34,57]. Gradient boosting is an ensemble technique that builds a strong predictive model by iteratively combining multiple weak models, often decision trees, to correct errors made by the previous models [31]. ...
This review article discusses the integration of artificial intelligence (AI) in assisted reproductive technology and provides key concepts to consider when introducing AI systems into reproductive medicine practices. The article highlights the various applications of AI in reproductive medicine and discusses whether to use commercial or in-house AI systems. This review also provides criteria for implementing new AI systems in the laboratory and discusses the factors that should be considered when introducing AI in the laboratory, including the user interface, scalability, training, support, follow-up, cost, ethics, and data quality. The article emphasises the importance of ethical considerations, data quality, and continuous algorithm updates to ensure the accuracy and safety of AI systems.
... This means that we are dealing with the decomposability of latent space (the representation is decomposable) and it is, therefore, intelligible to man. More specifically, Rudin et al. [8] discuss major challenges related to supervised and unsupervised disentanglement of neural networks to design an interpretable model and explain a black box. ...
... Let us suppose that an image can be generated by a set of semantically significant features such as colour, objects, shapes, etc. If these variation factors are captured separately in latent space and in an interpretable way, the image generation process becomes understandable and controllable by humans [8]. ...
... This group of DRLs depends on the learning scheme. Unsupervised disentanglement is when we do not know the concepts or the case, or when the concepts are numerous and we do not know how to parameterise them [8]. Interpretability can be achieved by automatic factorisation of latent representations. ...
Deep neural networks are widely used in computer vision for image classification, segmentation and generation. They are also often criticised as “black boxes” because their decision-making process is often not interpretable by humans. However, learning explainable representations that explicitly disentangle the underlying mechanisms that structure observational data is still a challenge. To further explore the latent space and achieve generic processing, we propose a pipeline for discovering the explainable directions in the latent space of generative models. Since the latent space contains semantically meaningful directions and can be explained, we propose a pipeline to fully resolve the representation of the latent space. It consists of a Dirichlet encoder, conditional deterministic diffusion, a group-swap and a latent traversal module. We believe that this study provides an insight into the advancement of research explaining the disentanglement of neural networks in the community.
... Rights reserved only summing coefficients from a fixed dictionary while Aug-Tree requires checking for the presence of keyphrases in an input. This allows for complete inspection of a model's decisionmaking process, unlike post hoc explanations, which are often unfaithful 11,15,16 . Across a variety of natural-language-processing datasets, both proposed Aug-imodels outperform their non-augmented counterparts. ...
... Additionally, transparent models tend to be dramatically more computationally efficient than LLMs. While transparent models can sometimes perform as well as black-box LLMs [11][12][13][14] , in many settings (such as natural language processing (NLP)), there is often a considerable gap between the performance of transparent models and black-box LLMs. ...
... At inference time, both are completely transparent and efficient: Aug-Linear requires only summing coefficients from a fixed dictionary while Aug-Tree requires checking for the presence of keyphrases in an input. This allows for complete inspection of a model's decisionmaking process, unlike post hoc explanations, which are often unfaithful 11,15,16 . ...
Recent large language models (LLMs), such as ChatGPT, have demonstrated remarkable prediction performance for a growing array of tasks. However, their proliferation into high-stakes domains and compute-limited settings has created a burgeoning need for interpretability and efficiency. We address this need by proposing Aug-imodels, a framework for leveraging the knowledge learned by LLMs to build extremely efficient and interpretable prediction models. Aug-imodels use LLMs during fitting but not during inference, allowing complete transparency and often a speed/memory improvement of greater than 1000x for inference compared to LLMs. We explore two instantiations of Aug-imodels in natural-language processing: Aug-Linear, which augments a linear model with decoupled embeddings from an LLM and Aug-Tree, which augments a decision tree with LLM feature expansions. Across a variety of text-classification datasets, both outperform their non-augmented, interpretable counterparts. Aug-Linear can even outperform much larger models, e.g. a 6-billion parameter GPT-J model, despite having 10,000x fewer parameters and being fully transparent. We further explore Aug-imodels in a natural-language fMRI study, where they generate interesting interpretations from scientific data.
... The recent spread of AI systems, especially Deep Learning systems, which are deemed black boxes due to their lack of explainability, has raised the popularity of eXplainable AI (XAI) [1], [2], [3]. XAI research focuses on the development of methods and techniques that aim at providing explanations of how these hardly interpretable AI systems make decisions [4], [5]. The research on XAI is multidisciplinary, rooted in cognitive and social science, and strongly related to humanmachine interaction, computational linguistics, etc. [6]. ...
... Rudin et al. [5] pointed out evaluation of explanations as a major challenge, among others, to be faced in the context of XAI. Accordingly, Hoffman et al. [7] introduced key concepts for measuring the quality of an XAI systemincluding the quality of explanations-derived from the integration of extensive research literature and psychometric assessments. ...
... For example, it is important to distinguish between independent 4 The taxonomy of interpretability issues proposed by Doshi-Velez and Kim [10] was already adopted in some publications [19], [20]. 5 To explain the task further, let us imagine that the AI system is a neural network classifier and the explanation is presented as a decision tree (DT). Then, this task would imply to use a DT to classify a given instance. ...
The assessment of explanations by humans presents a significant challenge within the context of Explainable and Trustworthy AI. This is attributed not only to the absence of universal metrics and standardized evaluation methods, but also to complexities tied to devising user studies that assess the perceived human comprehensibility of these explanations. To address this gap, we introduce a survey-based methodology for guiding the human evaluation of explanations. This approach amalgamates leading practices from existing literature and is implemented as an operational framework. This framework assists researchers throughout the evaluation process, encompassing hypothesis formulation, online user study implementation and deployment, and analysis and interpretation of collected data. The application of this framework is exemplified through two practical user studies.
... Lastly, DSs require being explained, and explanations for DSs are harder to compute than for DTs [22,2]. Furthermore, the paper indirectly proposes a practical solution to the abstract goal of intrinsic interpretability [44,41,46], where the classifier is itself the explanation. Indeed, the algorithm proposed in this paper offers a solution to deliver a classifier where the explanation can be extracted by manual inspection from the classifier. ...
... ¬x or x. (For the more complex examples, we will opt for the more compact notation, i.e. x.) DSs are distinguished from decision lists (DLs) [43] in that DLs are ordered, i.e. impose an order among the rules. Although DSs and DLs are generally considered to be interpretable [33,25,44,41,45,46], there is recent evidence to the contrary [22,38]. Besides the need for being explained, DSs exhibit a number of additional issues. ...
... In contrast with other algorithms for constructing decision sets proposed in the recent past [33,35,25,16,14,49,15,50,21], the algorithm proposed in this paper ensures that the resulting decision sets compute a total function, such that the condition of each rule is the explanation for the prediction when the rule fires. Given the existing proposals for intrinsic interpretability [44,41,46], the algorithm proposed in this paper offers a solution to deliver a classifier where the explanation is extracted, by inspection, from the classifier. The experiments demonstrate not only the ...
Recent work demonstrated that path explanation redundancy is ubiquitous in decision trees, i.e. most often paths in decision trees include literals that are redundant for explaining a prediction. The implication of this result is that decision trees must be explained. Nevertheless, there are applications of DTs where running an explanation algorithm is impractical. For example, in settings that are time or power constrained, running software algorithms for explaining predictions would be undesirable. Although the explanations for paths in DTs do not generally represent themselves a decision tree, this paper shows that one can construct a decision set from some of the decision tree explanations, such that the decision set is not only explained, but it also exhibits a number of properties that are critical for replacing the original decision tree.
... As a type of interpretable risk scoring model [17], scoring systems have been employed in practically every diagnostic area of medicine [18] since they offer quick and simple risk assessments of numerous serious medical conditions without the use of a computer [17]. Some traditional scoring systems, such as the Glasgow Coma Scale [19] first described in 1974, rely heavily on clinician's domain expertise. ...
... As a type of interpretable risk scoring model [17], scoring systems have been employed in practically every diagnostic area of medicine [18] since they offer quick and simple risk assessments of numerous serious medical conditions without the use of a computer [17]. Some traditional scoring systems, such as the Glasgow Coma Scale [19] first described in 1974, rely heavily on clinician's domain expertise. ...
... Scoring systems are linear classification models that require users to add, subtract and multiply a few numbers in order to make a prediction [17] and have been widely utilized in the field of clinical decisionmaking [26][27][28] for risk stratification due to their interpretability and transparency. They can also assist in correcting physicians' misestimations of the probability of medical outcomes, which may be rather common [29]. ...
Objective:
We propose FedScore, a privacy-preserving federated learning framework for scoring system generation across multiple sites to facilitate cross-institutional collaborations.
Materials and methods:
The FedScore framework includes five modules: federated variable ranking, federated variable transformation, federated score derivation, federated model selection and federated model evaluation. To illustrate usage and assess FedScore's performance, we built a hypothetical global scoring system for mortality prediction within 30 days after a visit to an emergency department using 10 simulated sites divided from a tertiary hospital in Singapore. We employed a pre-existing score generator to construct 10 local scoring systems independently at each site and we also developed a scoring system using centralized data for comparison.
Results:
We compared the acquired FedScore model's performance with that of other scoring models using the receiver operating characteristic (ROC) analysis. The FedScore model achieved an average area under the curve (AUC) value of 0.763 across all sites, with a standard deviation (SD) of 0.020. We also calculated the average AUC values and SDs for each local model, and the FedScore model showed promising accuracy and stability with a high average AUC value which was closest to the one of the pooled model and SD which was lower than that of most local models.
Conclusion:
This study demonstrates that FedScore is a privacy-preserving scoring system generator with potentially good generalizability.
... Especially in these areas, where the well of theoreticallydriven summary statistics is less deep than for selective sweeps, models that can learn new patterns and carry out these tasks without reliance on statistics provide an exciting way forward. Beyond achieving high performance at these tasks, we further propose that these models can help point us in useful directions: through advances in deep learning interpretation methods [23][24][25], we have the potential to conceive of new easy-to-implement summary statistics based on learned features. ...
... To address these questions, we first develop a very low-complexity CNN that we name "mini-CNN," which maintains high performance while allowing for more transparency into the classifier's inner workings. In addition, to interpret the more complex models, we draw on recent advances from the field of explainable AI [23][24][25], in particular using post-hoc interpretability methods built for visualizing and explaining feature importance for predictions after model training. Using these tools, we show empirically that typical preprocessing pipelines induce the CNN models for sweep detection to closely mimic handcrafted summary statistics; depending on how the data is presented to the models, their decision rules are particularly similar to Garud's H1, which typically performs at least as well at this classification task, or they pay closest attention to the number of segregating sites in a window, mimicking one of the earliest and simplest measures of genetic diversity, Watterson's theta. ...
A central challenge in population genetics is the detection of genomic footprints of selection. As machine learning tools including convolutional neural networks (CNNs) have become more sophisticated and applied more broadly, these provide a logical next step for increasing our power to learn and detect such patterns; indeed, CNNs trained on simulated genome sequences have recently been shown to be highly effective at this task. Unlike previous approaches, which rely upon human-crafted summary statistics, these methods are able to be applied directly to raw genomic data, allowing them to potentially learn new signatures that, if well-understood, could improve the current theory surrounding selective sweeps. Towards this end, we examine a representative CNN from the literature, paring it down to the minimal complexity needed to maintain comparable performance; this low-complexity CNN allows us to directly interpret the learned evolutionary signatures. We then validate these patterns in more complex models using metrics that evaluate feature importance. Our findings reveal that preprocessing steps, which determine how the population genetic data is presented to the model, play a central role in the learned prediction method. This results in models that mimic previously-defined summary statistics; in one case, the summary statistic itself achieves similarly high accuracy. For evolutionary processes that are less well understood than selective sweeps, we hope this provides an initial framework for using CNNs in ways that go beyond simply achieving high classification performance. Instead, we propose that CNNs might be useful as tools for learning novel patterns that can translate to easy-to-implement summary statistics available to a wider community of researchers.
... The problem of determining which features are actually the most informative ones for a certain analysis is known as the feature (or variable) selection problem [29]. Selecting the appropriate subset of variables from a given set is crucial to obtain interpretable machine learning models [40] and to improve their predictive power, since overfitting is reduced. Mathematical optimization plays an important role in this task as a powerful tool to model the feature selection problem subject to additional requirements provided by the specific nature of the data analysis at hand and also to develop algorithms which can obtain a good solution in a reasonable time. ...
... However, as in the variable selection methods for linear regression using convex penalties, these approaches impose a strong unintended bias on the regression coefficients. These limitations are exposed in [40], in which two main challenges regarding how to control the interpretability of the generalized additive models (e.g. by means of inducing sparsity in the model) and their potential to troubleshoot complex data sets are stated. In this paper, we partially address the first challenge by stating the feature selection problem in AMs as a cardinality-constrained model using mixed-integer quadratic optimization and providing the model with the flexibility of selecting each variable as linear, non-linear, or both. ...
Feature selection is a recurrent research topic in modern regression analysis, which strives to build interpretable models, using sparsity as a proxy, without sacrificing predictive power. The best subset selection problem is central to this statistical task: it has the goal of identifying the subset of covariates of a given size that provides the best fit in terms of an empirical loss function. In this work, we address the problem of feature selection in additive regression models under a mathematical optimization lens. Penalized splines (P −splines) are used to estimate the smooth functions involved in the regression equation, which allow us to state the feature selection problem as a cardinality-constrained mixed-integer quadratic program (MIQP) in terms of both linear and non-linear covariates. To strenghten this MIQP formulation, we develop tight bounds for the regression coefficients. A matheuristic approach, which encompasses the use of a warm-start solution, the MIQP formulation and the large neighborhood search meta-heuristic paradigm, is proposed to handle larger instances of the feature selection problem. The performance of the exact and the matheuristic approaches are compared in simulated data. Furthermore, our matheuristic is compared with other state-of-the-art methodologies in feature selection in additive models in both simulated and real-world data. We show that the stated approach is competitive in terms of predictive power and in the selection of the correct subset of covariates. A public Python library is available with all the implementations of the methodologies developed in this paper.
... Beside accuracy, interpretability is an essential quality and a key factor of trust for an algorithm. A non-interpretable model, also called a black box model, can "predict the right answer for the wrong reason" [83]. As a result, it is questionable whether a black box model could generalize beyond the training dataset [83]. ...
... A non-interpretable model, also called a black box model, can "predict the right answer for the wrong reason" [83]. As a result, it is questionable whether a black box model could generalize beyond the training dataset [83]. A prime example of this caveat is Al-phaFold2 [44], a black box model that achieved near-experimental accuracy on the CASP14 protein folding challenge [49] but is unable to predict the impact of structure-disrupting mutations, which are frequently associated with protein aggregation, misfolding, and dysfunction [11,75]. ...
Accurate binding affinity prediction is crucial to structure-based drug design. Recent work used computational topology to obtain an effective representation of protein-ligand interactions. Although persistent homology encodes geometric features, previous works on binding affinity prediction using persistent homology employed uninterpretable machine learning models and failed to explain the underlying geometric and topological features that drive accurate binding affinity prediction. In this work, we propose a novel, interpretable algorithm for protein-ligand binding affinity prediction. Our algorithm achieves interpretability by an effective embedding of distances across bipartite matchings of the protein and ligand atoms into real-valued functions by summing Gaussians centered at features constructed by persistent homology. We name these functions \textit{internuclear persistent contours (IPCs)}. Next, we introduce \textit{persistence fingerprints}, a vector with 10 components that sketches the distances of different bipartite matching between protein and ligand atoms, refined from IPCs. Let the number of protein atoms in the protein-ligand complex be $n$, number of ligand atoms be $m$, and $\omega \approx 2.4$ be the matrix multiplication exponent. We show that for any $0 < \epsilon < 1$, after an $O(mn \log(mn))$ preprocessing procedure, we can compute an $\epsilon$-accurate approximation to the persistence fingerprint in $O(m \log^{6\omega} (m/\epsilon))$ time, independent of protein size. This is an improvement in time complexity by a factor of $O((m+n)^3)$ over any previous binding affinity prediction that uses persistent homology. We show that the representational power of persistence fingerprint generalizes to protein-ligand binding datasets beyond the training dataset. Then, we introduce \textit{PATH}, Predicting Affinity Through Homology, an interpretable, small ensemble of shallow regression trees for binding affinity prediction from persistence fingerprints. We show that despite using 1,400-fold fewer features, PATH has comparable performance to a previous state-of-the-art binding affinity prediction algorithm that uses persistent homology features. Moreover, PATH has the advantage of being interpretable. Finally, we visualize the features captured by persistence fingerprint for variant HIV-1 protease complexes and show that persistence fingerprint captures binding-relevant structural mutations. The source code for PATH is released open-source as part of the OSPREY protein design software package.
... In recent years, there has been growing interest in interpretable Machine Learning (ML) models (Rudin et al., 2022). Decision trees are among the most popular Supervised ML tools used for classification tasks. ...
... As a second contribution, we aim to reduce the number of features used in each split to enhance the interpretability of the model itself. Indeed, for tabular data, sparsity is a core component of interpretability (Rudin et al., 2022), and having fewer features selected at each branching node allows the end user to identify the key factors affecting the classification of the samples. ...
In recent years, there has been growing attention to interpretable machine learning models which can give explanatory insights on their behaviour. Thanks to their interpretability, decision trees have been intensively studied for classification tasks and, due to the remarkable advances in mixed integer programming (MIP),
various approaches have been proposed to formulate the problem of training an Optimal Classification Tree (OCT) as a MIP model. We present a novel mixed integer quadratic formulation for the OCT problem, which exploits the generalization capabilities of Support Vector Machines for binary classification. Our model, denoted as Margin Optimal Classification Tree (MARGOT), encompasses maximum margin multivariate
hyperplanes nested in a binary tree structure. To enhance the interpretability of our approach, we analyse two alternative versions of MARGOT, which include feature selection constraints inducing sparsity of the hyperplanes’ coefficients. First, MARGOT has been tested on non-linearly separable synthetic datasets in a 2-dimensional feature space to provide a graphical representation of the maximum margin approach. Finally, the proposed models have been tested on benchmark datasets from the UCI repository. The MARGOT formulation turns out to be easier to solve than other OCT approaches, and the generated tree better generalizes on new observations. The two interpretable versions effectively select the most relevant features, maintaining good prediction quality.
... There has been a lot of focus on this issue since the landmark publication by Szegedy et al. [3], which demonstrated that the most cutting-edge neural networks are susceptible to adversarial attacks. The research has resulted in investigations into several adversarial threat models and situations [8], as well as attacks that are both computationally and perturbational efficient [9], etc. ...
... : Update x * t+1 by applying the sign gradient as x * t+1 = x * t + α · sign(g t+1 ); 7: end for 8: ...
Adversarial attacks exploit vulnerabilities or weaknesses in the model’s decision-making process to generate inputs that appear benign to humans but can lead to incorrect or unintended outputs from the model. Neural networks (NNs) are widely used for aerial detection, and increased usage has highlighted the vulnerability of DNNs to adversarial cases intentionally designed to mislead them. The majority of adversarial attacks now in use can only rarely deceive a black-box model. We employ the fast gradient sign technique (FGSM) to immediately enhance the position of an adversarial area to identify the target. We employ two open datasets in extensive experiments; however, the findings demonstrate that, on average, only 400 queries may successfully perturb at least one erroneous class in most of the photos in the test dataset. The proposed method can be used for both untargeted and targeted attacks, leading to incredible query efficiency in both scenarios. The experiment manipulates input images using gradients or noise to generate misclassified outputs. It is implemented in Python using the TensorFlow framework. The experiment optimizes performance by using an algorithm with an initial learning rate of 0.1 and adjusting the learning rate based on the number of training samples using different epoch values. Compared to other studies, our technique outperforms them in crafting adversaries and provides high accuracy. Moreover, this technique works effectively, needs a few lines of code to be implemented, and functions as a solid base for upcoming black-box attacks.
... These systems often act as black-boxes that lack of interpretability. Making Machine Learning systems trustworthy has become imperative, and interpretability, robustness, and fairness are often essential requirements for deployment (European Commission, 2020;Goodman & Flaxman, 2017;Rudin, Chen, Chen, Huang, Semenova, & Zhong, 2022). This paper is devoted to enhancing the interpretability of blackbox classifiers. ...
... Rudin asserts that interpretability is a domain-specific notion, which is why there cannot be a satisfactory all-purpose definition (2019; see also Rudin et al. 2022). However, a common denominator in interpretable models is that they are constrained in some wayfor example, by way of monotonicity, additivity, causality, sparsity, or by incorporating domain knowledge (Rudin 2019, 206). ...
This paper develops an account of the opacity problem in medical machine learning (ML). Guided by pragmatist assumptions, I argue that opacity in ML models is problematic insofar as it potentially undermines the achievement of two key purposes: ensuring generalizability and optimizing clinician–machine decision-making. Three opacity amelioration strategies are examined, with explainable artificial intelligence (XAI) as the predominant approach, challenged by two revisionary strategies in the form of reliabilism and the interpretability by design. Comparing the three strategies, I argue that interpretability by design is most promising to overcome opacity in medical ML. Looking beyond the individual opacity amelioration strategies, the paper also contributes to a deeper understanding of the problem space and the solution space regarding opacity in medical ML.
... Both explainability and interpretability of ML models have been lauded as essential for trustworthy AI and listed as criteria for trustworthy AI systems (e.g., Tabassi, 2023). Rudin et al. (2022) have, for example, offered five principles for creating a predictive AI model that is not a black box. In their terms, interpretability is achieved to the extent that the AI obeys a domain-specific set of constraints that allow it to be more easily understood by humans (Principle 1). ...
Demands to manage the risks of artificial intelligence (AI) are growing. These demands and the government standards arising from them both call for trustworthy AI. In response, we adopt a convergent approach to review, evaluate, and synthesize research on the trust and trustworthiness of AI in the environmental sciences and propose a research agenda. Evidential and conceptual histories of research on trust and trustworthiness reveal persisting ambiguities and measurement shortcomings related to inconsistent attention to the contextual and social dependencies and dynamics of trust. Potentially underappreciated in the development of trustworthy AI for environmental sciences is the importance of engaging AI users and other stakeholders, which human–AI teaming perspectives on AI development similarly underscore. Co‐development strategies may also help reconcile efforts to develop performance‐based trustworthiness standards with dynamic and contextual notions of trust. We illustrate the importance of these themes with applied examples and show how insights from research on trust and the communication of risk and uncertainty can help advance the understanding of trust and trustworthiness of AI in the environmental sciences.
... Machine learning techniques have become a potentially valuable tool for data detection by generating data-based predictions (Kleinberg et al., 2015;Mullainathan and Spiess, 2017). Though they have been successfully used, there are still drawbacks including a lack of interpretability and data quality sensitivity, and these issues can be addressed by domain knowledge to some extent (Du et al., 2019;Rudin et al., 2022). For example, in the energy domain, Hu et al. (2019) detected faults from measured energy data by integrating domain knowledge with machine learning model; Manojlović et al. (2022) improved data quality and model performance by incorporating domain knowledge into their dataflow management before evaluating energy efficiency parameters. ...
... A likely reason for this may be that these algorithms are simple and easier to implement and understand, as well as more interpretable compared to deep learning methods [63]. Interpretation of deep learning can be challenging because the steps that are taken to arrive at the final analytical output are not always as clear as those used in more traditional methods [63][64][65]. In addition, rule-based and traditional algorithms are more useful for smaller datasets with few features as these algorithms do not require massive amounts of data that are necessary for the development and successful implementation of machine learning. ...
Background
Extracting information from free texts using natural language processing (NLP) can save time and reduce the hassle of manually extracting large quantities of data from incredibly complex clinical notes of cancer patients. This study aimed to systematically review studies that used NLP methods to identify cancer concepts from clinical notes automatically.
Methods
PubMed, Scopus, Web of Science, and Embase were searched for English language papers using a combination of the terms concerning “Cancer”, “NLP”, “Coding”, and “Registries” until June 29, 2021. Two reviewers independently assessed the eligibility of papers for inclusion in the review.
Results
Most of the software programs used for concept extraction reported were developed by the researchers (n = 7). Rule-based algorithms were the most frequently used algorithms for developing these programs. In most articles, the criteria of accuracy (n = 14) and sensitivity (n = 12) were used to evaluate the algorithms. In addition, Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT) and Unified Medical Language System (UMLS) were the most commonly used terminologies to identify concepts. Most studies focused on breast cancer (n = 4, 19%) and lung cancer (n = 4, 19%).
Conclusion
The use of NLP for extracting the concepts and symptoms of cancer has increased in recent years. The rule-based algorithms are well-liked algorithms by developers. Due to these algorithms' high accuracy and sensitivity in identifying and extracting cancer concepts, we suggested that future studies use these algorithms to extract the concepts of other diseases as well.
... Many challenges to the applications of AI/ML approaches in clinical practice and clinical trials still remain. These include the requirement for relatively large and well-labeled training datasets 15 , uneven image quality and specimen collection methods necessitating extensive standardization and normalization of data 16,17 , hardware limitations encountered when processing large high-resolution images 18 , as well as ethical and privacy concerns arising from interaction with human subjects 19 . ...
Background and Aims
Histologic disease activity in Inflammatory Bowel Disease (IBD) is associated with clinical outcomes and is an important endpoint in drug development. We developed deep learning models for automating histological assessments in IBD.
Methods
Histology images of intestinal mucosa from phase 2 and phase 3 clinical trials in Crohn’s disease (CD) and Ulcerative Colitis (UC) were used to train artificial intelligence (AI) models to predict the Global Histology Activity Score (GHAS) for CD and Geboes histopathology score for UC. Three AI methods were compared. AI models were evaluated on held-back testing sets and model predictions were compared against an expert central reader and five independent pathologists.
Results
The model based on multiple instance learning and the attention mechanism (SA-AbMILP) demonstrated the best performance among competing models. AI modeled GHAS and Geboes sub-grades matched central readings with moderate to substantial agreement, with accuracies ranging from 65% to 89%. Furthermore, the model was able to distinguish the presence and absence of pathology across four selected histological features with accuracies for colon, in both CD and UC, ranging from 87% to 94% and, for CD ileum, ranging from 76% to 83%. For both CD and UC, and across anatomical compartments (ileum and colon) in CD, comparable accuracies against central readings were found between the model assigned scores and scores by an independent set of pathologists.
Conclusions
Deep learning models based upon GHAS and Geboes scoring systems were effective at distinguishing between the presence and absence of IBD microscopic disease activity.
... For image data, where neural networks are the only technique that performs well currently, interpretable neural networks perform as well as black box neural networks. Thus, for either tabular data or images, interpretable models are generally as accurate as the best of the black box models when applied to benchmark datasets (22). ...
One of the most troubling trends in criminal investigations is the growing use of “black box” technology, in which law enforcement rely on artificial intelligence (AI) models or algorithms that are either too complex for people to understand or they simply conceal how it functions. In criminal cases, black box systems have proliferated in forensic areas such as DNA mixture interpretation, facial recognition, and recidivism risk assessments. The champions and critics of AI argue, mistakenly, that we face a catch 22: While black box AI is not understandable by people, they assume that it produces more accurate forensic evidence. In this Article, we question this assertion, which has so powerfully affected judges, policymakers, and academics. We describe a mature body of computer science research showing how “glass box” AI—designed to be interpretable—can be more accurate than black box alternatives. Indeed, black box AI performs predictably worse in settings like the criminal system. Debunking the black box performance myth has implications for forensic evidence, constitutional criminal procedure rights, and legislative policy. Absent some compelling—or even credible—government interest in keeping AI as a black box, and given the constitutional rights and public safety interests at stake, we argue that a substantial burden rests on the government to justify black box AI in criminal cases. We conclude by calling for judicial rulings and legislation to safeguard a right to interpretable forensic AI.
... Explaining the behaviour of AI systems is an issue of major significance in the perspective of trustworthy AI. Thus, recent years have seen a remarkable boom in work aimed at verifying AI systems and explaining the outputs they generate (see for instance [18,19,21,24,28,31,34,41,1,9,39]). ...
We define contrastive explanations that are suited to tree-based classifiers. In our framework, contrastive explanations are based on the set of (possibly non-independent) Boolean characteristics used by the classifier and are at least as general as contrastive explanations based on the set of characteristics of the instances considered at start. We investigate the computational complexity of computing contrastive explanations for Boolean classifiers (including tree-based ones), when the Boolean conditions used are not independent. Finally, we present and evaluate empirically an algorithm for computing minimum-size contrastive explanations for random forests.
... Researchers often categorise AI systems based on interpretability [6,56], contrasting non-interpretable black box systems such as neural networks against interpretable white box systems such as decision trees. Interpretability has a crucial influence on the design choices of XAI methods: using white-box models, we can more easily guarantee verifiability and causality. ...
The European Union has proposed the Artificial Intelligence Act which introduces detailed requirements of transparency for AI systems. Many of these requirements can be addressed by the field of explainable AI (XAI), however, there is a fundamental difference between XAI and the Act regarding what transparency is. The Act views transparency as a means that supports wider values, such as accountability, human rights, and sustainable innovation. In contrast, XAI views transparency narrowly as an end in itself, focusing on explaining complex algorithmic properties without considering the socio-technical context. We call this difference the “transparency gap”. Failing to address the transparency gap, XAI risks leaving a range of transparency issues unaddressed. To begin to bridge this gap, we overview and clarify the terminology of how XAI and European regulation – the Act and the related General Data Protection Regulation (GDPR) – view basic definitions of transparency. By comparing the disparate views of XAI and regulation, we arrive at four axes where practical work could bridge the transparency gap: defining the scope of transparency, clarifying the legal status of XAI, addressing issues with conformity assessment, and building explainability for datasets.
... Answers to such XAI queries can be used by the user of the ML model to determine whether the model is trustable enough (or not), by confronting those answers to the own expectations of the user about the model. Many work has been devoted for the past few years in this direction (see e.g., [1,28,33,18,36,29,27,32]). The issue of correcting the system's mistakes has received less attention, probably because it is more tricky. ...
We elaborate on the notion of rectification of a classifier Σ based on Boolean features, introduced in [10]. The purpose is to determine how to modify Σ when the way it classifies a given instance is considered incorrect since it conflicts with some expert knowledge T. Given Σ and T, postulates characterizing the way Σ must be changed into a new classifier Σ ⋆ T that complies with T were presented. We focus here on the specific case of binary classifiers, i.e., there is a single target concept, and any instance is classified either as positive (an element of the concept), or as negative (an element of the complementary concept). In this specific case, our main contribution is twofold: (1) we show that there is a unique rectification operator ⋆ satisfying the postulates, and (2) when Σ and T are Boolean circuits, we show how a classification circuit equivalent to Σ ⋆ T can be computed in time linear in the size of Σ and T; when Σ is a decision tree (resp. a random forest, a boosted tree) and T is a decision tree, a decision tree (resp. a random forest, a boosted tree) equivalent to Σ ⋆ T can be computed in time polynomial in the size of Σ and T.
... The identification of operational (as well as design) issues using digital replicas of objects/processes/services is part of current research in several fields and a multiplicity of "digital twin" definitions are present across different studies [13], in which sometimes the difference between model and digital twin is blurred [14]. Further, this topic is attracting a lot of attention also in building and energy sectors [15], due to the increasing use of Artificial Intelligence (AI) and Machine Learning (ML) tools; however, while sophisticated ML techniques such as deep learning have been effective in multiple applications [16], they lack interpretability [17], which is today an open challenge [18]. As show by Chen et al. [19] sophisticated machine learning approaches used in building energy management employ, in most of the cases, post-hoc techniques such as LIME and SHAP [20] to interpret their results, but it is very difficult to inspect the model algorithmic logic in a simple and transparent way. ...
... The research enhances the accessibility and engagement with these cultural artifacts. In [24] employs social network analysis to analyze the cultural influences on ancient Chinese stone carving decoration. It explores the connections, collaborations, and exchanges between artists and workshops to understand the diffusion of stylistic elements. ...
In recent years, advancements in data analysis techniques and deep learning algorithms have revolutionized the field of art and cultural studies. Ancient Chinese stone carving decoration holds significant historical and cultural value, reflecting the artistic and stylistic evolution of different periods. This paper explored the Weighted Long Short-Term Memory Deep Learning (WLSTM – DL) evolution and stylistic characteristics of ancient Chinese stone carving decoration through the application of image visualization techniques combined with a Long Short-Term Memory (LSTM) time-series deep learning architecture. The WLSTM-DL model uses the optimized feature selection with the grasshopper optimization for the feature extraction and selection. By analyzing a comprehensive dataset of stone carving images from different periods, the WLSTM-DL model captures the temporal relationships and patterns in the evolution of stone carving decoration. The model utilizes LSTM, a specialized deep-learning architecture for time-series data, to uncover stylistic characteristics and identify significant changes over time. The findings of this study provide valuable insights into the evolution and stylistic development of ancient Chinese stone carving decoration. The application of image visualization techniques and the WLSTM-DL model showcase the potential of data analysis and deep learning in uncovering hidden narratives and understanding the intricate details of ancient artworks.
... However, the literature has also reported some controversial/unfair decisions made with AI/ML algorithms when, e.g., assessing the risk of potential recidivism or making social benefit allocations [Rudin, 2019]. This, together with the need of users (e.g., physicians, judges, civil servants, citizens) to understand why the model made a decision, calls for enhancing the transparency of Machine Learning algorithms [Blanquero et al., 2020, European Commission, 2020, Goodman and Flaxman, 2017, Panigutti et al., 2023, Rudin et al., 2022. In this paper, we contribute to this stream of literature enhancing the transparency of tree ensembles. ...
Tree ensembles are one of the most powerful methodologies in Machine Learning. In this paper, we investigate how to make tree ensembles more flexible to incorporate by design explainability and fairness. While explainability helps the user understand the key features that play a role in the classification task, with fairness we ensure that the ensemble does not discriminate against a group of observations that share a sensitive attribute. We propose a Mixed Integer Linear Optimization formulation to train an ensemble of trees that apart from minimizing the misclassification error, controls for sparsity as well as the accuracy in the sensitive group. Our formulation is scalable in the number of observations since its number of binary decision variables is independent of the number of observations. In our numerical results, we show that for standard datasets used in the 1 fairness literature, we can dramatically enhance the fairness of the benchmark, namely the popular Random Forest, while using only a few features, all without damaging the misclassification error.
... Accordingly, prototypebased regression models inherit the properties known from VQ methods and its variations. Main advantages given by VQ are interpretability and robustness [3,4,5], whereas black box models like multi-layer-perceptrons (MLP) are in need of explainability. Yet, techniques to gain insight into a black box model in terms of regression were recently proposed by [6] in which the restructuring approach involves a reference point, which can in this regard be interpreted as a prototype in the linear layer fulfilling certain rules. ...
... On the other hand, Breiman (2001) and much work from the machine learning community seem to reduce aims of modeling to prediction quality as all-dominating issue, which neglects the role of models for communication and building understanding (see the Section 2.3), even though it is probably valid to state that prediction quality is very often at least implied by the aim of analysis. Recently, this issue has been acknowledged also in the machine learning community, and there is a raising number of publications on the interpretability of the outcomes of machine learning algorithms (Rudin et al. 2022). ...
... Rudin et al. [74] Identifies the top 10 challenges in interpretable ML along with their comprehensive background and solution approach. ...
Recent years have seen a tremendous growth in Artificial Intelligence (AI)-based methodological development in a broad range of domains. In this rapidly evolving field, large number of methods are being reported using machine learning (ML) and Deep Learning (DL) models. Majority of these models are inherently complex and lacks explanations of the decision making process causing these models to be termed as 'Black-Box'. One of the major bottlenecks to adopt such models in mission-critical application domains, such as banking, e-commerce, healthcare, and public services and safety, is the difficulty in interpreting them. Due to the rapid proleferation of these AI models, explaining their learning and decision making process are getting harder which require transparency and easy predictability. Aiming to collate the current state-of-the-art in interpreting the black-box models, this study provides a comprehensive analysis of the explainable AI (XAI) models. To reduce false negative and false positive outcomes of these back-box models, finding flaws in them is still difficult and inefficient. In this paper, the development of XAI is reviewed meticulously through careful selection and analysis of the current state-of-the-art of XAI research. It also provides a comprehensive and in-depth evaluation of the XAI frameworks and their efficacy to serve as a starting point of XAI for applied and theoretical researchers. Towards the end, it highlights emerging and critical issues pertaining to XAI research to showcase major, model-specific trends for better explanation, enhanced transparency, and improved prediction accuracy.
Aiming to discuss artificial intelligence (AI) fundamentals, this chapter was written with a slightly diverse approach, doing that by observing some recent market movements where AI was really or potentially applied. This way to express was chosen because it offers an alternative of reflection and essential discussion of the topic, with the objective as to produce a complementary level of understanding to the huge technical, social, and legal production already available for immediate consultation through Internet. In this chapter, a brief historical path for AI is analyzed, compared to other tech and market efforts, presenting main fundamentals and concepts in this story, developing a deeper analysis in the following study case approaching. Through these cases, concepts and artificial intelligence relations and contributions are researched, completing the initial intended level for a theoretical (and practical) background. Artificial intelligence is here to stay. And evolve. To where, we still do not know.
Despite progress in the field, significant parts of current XAI research are still not on solid conceptual, ethical, or methodological grounds. Unfortunately, these unfounded parts are not on the decline but continue to grow. Many explanation techniques are still proposed without clarifying their purpose. Instead, they are advertised with ever more fancy-looking heatmaps or only seemingly relevant benchmarks. Moreover, explanation techniques are motivated with questionable goals, such as building trust, or rely on strong assumptions about the ’concepts’ that deep learning algorithms learn. In this paper, we highlight and discuss these and other misconceptions in current XAI research. We also suggest steps to make XAI a more substantive area of research.
The shear wave velocity (Vs) is significant for quantitative seismic interpretation. Although numerous studies have proved the effectiveness of the machine learning method in estimating the Vs using well-logging parameters, the real-world application is still hindered because of the black-box nature of machine learning models. With the rapid development of the interpretable machine learning (ML) technique, the drawback of ML can be overcome by various interpretation methods. This study applies the Light Gradient Boosting Machine (LightGBM) to predict the Vs of a carbonate reservoir and uses the Shapley Additive Explanations (SHAP) to interpret the model. The application of ML in Vs estimation normally involves using conventional well-log data that are highly correlated with Vs to train the model. To expand the model’s applicability in wells that lack essential logs, such as the density and neutron logs, we introduce three geologically important features, temperature, pressure, and formation, into the model. The LightGBM model is tuned by the automatic hyperparameter optimization framework; the result is compared with the Xu-Payne rock physics model and four machine learning models tuned with the same process. The results show that the LightGBM model can fit the training data and provide accurate predictions in the test well. The model outperforms the rock physics model and other ML models in both accuracy and training time. The SHAP analysis provides a detailed explanation of the contribution of each input variable to the model and demonstrates the variation of feature contribution in different reservoir conditions. Moreover, the validity of the LightGBM model is further proved by the consistency of the deduced information from feature dependency with the geological understanding of the carbonate formation. The study demonstrates that the newly added features can effectively improve model performance, and the importance of the input feature is not necessarily related to its correlation with Vs
Building generalizable AI models is one of the primary challenges in the healthcare domain. While radiologists rely on generalizable descriptive rules of abnormality, Neural Network (NN) models suffer even with a slight shift in input distribution (e.g., scanner type). Fine-tuning a model to transfer knowledge from one domain to another requires a significant amount of labeled data in the target domain. In this paper, we develop an interpretable model that can be efficiently fine-tuned to an unseen target domain with minimal computational cost. We assume the interpretable component of NN to be approximately domain-invariant. However, interpretable models typically underperform compared to their Blackbox (BB) variants. We start with a BB in the source domain and distill it into a mixture of shallow interpretable models using human-understandable concepts. As each interpretable model covers a subset of data, a mixture of interpretable models achieves comparable performance as BB. Further, we use the pseudo-labeling technique from semi-supervised learning (SSL) to learn the concept classifier in the target domain, followed by fine-tuning the interpretable models in the target domain. We evaluate our model using a real-life large-scale chest-X-ray (CXR) classification dataset. The code is available at: https://github.com/batmanlab/MICCAI-2023-Route-interpret-repeat-CXRs.
This paper proposes a new framework for learning a rule ensemble model that is both accurate and interpretable. A rule ensemble is an interpretable model based on the linear combination of weighted rules. In practice, we often face the trade-off between the accuracy and interpretability of rule ensembles. That is, a rule ensemble needs to include a sufficiently large number of weighted rules to maintain its accuracy, which harms its interpretability for human users. To avoid this trade-off and learn an interpretable rule ensemble without degrading accuracy, we introduce a new concept of interpretability, named local interpretability, which is evaluated by the total number of rules necessary to express individual predictions made by the model, rather than to express the model itself. Then, we propose a regularizer that promotes local interpretability and develop an efficient algorithm for learning a rule ensemble with the proposed regularizer by coordinate descent with local search. Experimental results demonstrated that our method learns rule ensembles that can explain individual predictions with fewer rules than the existing methods, including RuleFit, while maintaining comparable accuracy.
Inequalities and injustices are thorny issues in liberal societies, manifesting in forms such as the gender-pay gap; sentencing discrepancies among Black, Hispanic, and White defendants; and unequal medical-resource distribution across ethnicities. One cause of these inequalities is implicit social bias-unconsciously formed associations between social groups and attributions such as "nurturing," "lazy," or "uneducated." One strategy to counteract implicit and explicit human biases is delegating crucial decisions, such as how to allocate benefits, resources, or opportunities, to algorithms. Algorithms, however, are not necessarily impartial and objective. Although they can detect and mitigate human biases, they can also perpetuate and even amplify existing inequalities and injustices. We explore how a philosophical thought experiment, Rawls's "veil of ignorance," and a psychological phenomenon, deliberate ignorance, can help shield individuals, institutions, and algorithms from biases. We discuss the benefits and drawbacks of methods for shielding human and artificial decision makers from potentially biasing information. We then broaden our discussion beyond the issues of bias and fairness and turn to a research agenda aimed at improving human judgment accuracy with the assistance of algorithms that conceal information that has the potential to undermine performance. Finally, we propose interdisciplinary research questions.
Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of reinforcement learning (RL) agents in sequential decision-making settings. Equipped with this information, practitioners can better understand important questions about RL agents (especially those deployed in the real world), such as what the agents will do and why. Despite increased interest, there exists a gap in the literature for organizing the plethora of papers — especially in a way that centers the sequential decision-making nature of the problem. In this survey, we propose a novel taxonomy for organizing the XRL literature that prioritizes the RL setting. We propose three high-level categories: feature importance, learning process and Markov decision process, and policy-level. We overview techniques according to this taxonomy, highlighting challenges and opportunities for future work. We conclude by using these gaps to motivate and outline a roadmap for future work.
The use of healthcare systems has a big impact on people's well-being. To create efficient healthcare systems, new models are always being developed. The quick growth of the use of such models in the medical disciplines has created great possibilities for the development of new applications. However, the development of quick, precise, effective models suitable for medical applications still faces significant hurdles. In this chapter, the feasibility of applying the quantum support vector classifier algorithm (QSVC) is evaluated and tested on medical datasets. Ten datasets obtained from the UCI machine learning repository were adopted for this study. The experimental results revealed that the proposed intelligent model based on the QSVC obtained very promising results. The results showed the high classification outcomes of QSVC compared with state of art models. These high classification results can offer technical assistance for the enhancement of medical data classification.KeywordsIntelligent modelQuantum computingQSVCArtificial intelligenceMedical data
Machine learning (ML) systems often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification in ML pipelines as a key reason for these failures. An ML pipeline is the full procedure followed to train and validate a predictor. Such a pipeline is underspecified when it can return many distinct predictors with equivalently strong test performance. Underspecification is common in modern ML pipelines that primarily validate predictors on held-out data that follow the same distribution as the training data. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We provide evidence that underspecfication has substantive implications for practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.
Interpretability in machine learning models is important in high-stakes decisions such as whether to order a biopsy based on a mammographic exam. Mammography poses important challenges that are not present in other computer vision tasks: datasets are small, confounding information is present and it can be difficult even for a radiologist to decide between watchful waiting and biopsy based on a mammogram alone. In this work we present a framework for interpretable machine learning-based mammography. In addition to predicting whether a lesion is malignant or benign, our work aims to follow the reasoning processes of radiologists in detecting clinically relevant semantic features of each image, such as the characteristics of the mass margins. The framework includes a novel interpretable neural network algorithm that uses case-based reasoning for mammography. Our algorithm can incorporate a combination of data with whole image labelling and data with pixel-wise annotations, leading to better accuracy and interpretability even with a small number of images. Our interpretable models are able to highlight the classification-relevant parts of the image, whereas other methods highlight healthy tissue and confounding information. Our models are decision aids—rather than decision makers—and aim for better overall human–machine collaboration. We do not observe a loss in mass margin classification accuracy over a black box neural network trained on the same data.
The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50--100x across a range of problems in computational physics.
Decision trees have been a very popular class of predictive models for decades due to their interpretability and good performance on categorical features. However, they are not always robust and tend to overfit the data. Additionally, if allowed to grow large, they lose interpretability. In this paper, we present a mixed integer programming formulation to construct optimal decision trees of a prespecified size. We take the special structure of categorical features into account and allow combinatorial decisions (based on subsets of values of features) at each node. Our approach can also handle numerical features via thresholding. We show that very good accuracy can be achieved with small trees using moderately-sized training sets. The optimization problems we solve are tractable with modern solvers.
Classification and regression trees, as well as their variants, are off-the-shelf methods in Machine Learning. In this paper, we review recent contributions within the Continuous Optimization and the Mixed-Integer Linear Optimization paradigms to develop novel formulations in this research area. We compare those in terms of the nature of the decision variables and the constraints required, as well as the optimization algorithms proposed. We illustrate how these powerful formulations enhance the flexibility of tree models, being better suited to incorporate desirable properties such as cost-sensitivity, explainability, and fairness, and to deal with complex data, such as functional data.
With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.
Decision trees are a popular choice for providing explainable machine learning, since they make explicit how different features contribute towards the prediction. We apply tools from constraint satisfaction to learn optimal decision trees in the form of sparse k-CNF (Conjunctive Normal Form) rules. We develop two methods offering different trade-offs between accuracy and computational complexity: one offline method that learns decision trees using the entire training dataset and one online method that learns decision trees over a local subset of the training dataset. This subset is obtained from training examples near a query point. The developed methods are applied on a number of datasets both in an online and an offline setting. We found that our methods learn decision trees which are significantly more accurate than those learned by existing heuristic approaches. However, the global decision tree model tends to be computationally more expensive compared to heuristic approaches. The online method is faster to train and finds smaller decision trees with an accuracy comparable to that of the k-nearest-neighbour method.
Variable importance is central to scientific studies, including the social sciences and causal inference, healthcare and other domains. However, current notions of variable importance are often tied to a specific predictive model. This is problematic: what if there were multiple well-performing predictive models, and a specific variable is important to some of them but not to others? In that case, we cannot tell from a single well-performing model if a variable is always important, sometimes important, never important or perhaps only important when another variable is not important. Ideally, we would like to explore variable importance for all approximately equally accurate predictive models within the same model class. In this way, we can understand the importance of a variable in the context of other variables, and for many good models. This work introduces the concept of a variable importance cloud, which maps every variable to its importance for every good predictive model. We show properties of the variable importance cloud and draw connections to other areas of statistics. We introduce variable importance diagrams as a projection of the variable importance cloud into two dimensions for visualization purposes. Experiments with criminal justice, marketing data and image classification tasks illustrate how variables can change dramatically in importance for approximately equally accurate predictive models.
What does a neural network encode about a concept as we traverse through the layers? Interpretability in machine learning is undoubtedly important, but the calculations of neural networks are very challenging to understand. Attempts to see inside their hidden layers can be misleading, unusable or rely on the latent space to possess properties that it may not have. Here, rather than attempting to analyse a neural network post hoc, we introduce a mechanism, called concept whitening (CW), to alter a given layer of the network to allow us to better understand the computation leading up to that layer. When a concept whitening module is added to a convolutional neural network, the latent space is whitened (that is, decorrelated and normalized) and the axes of the latent space are aligned with known concepts of interest. By experiment, we show that CW can provide us with a much clearer understanding of how the network gradually learns concepts over layers. CW is an alternative to a batch normalization layer in that it normalizes, and also decorrelates (whitens), the latent space. CW can be used in any layer of the network without hurting predictive performance.
Artificial intelligence (AI) brings forth many opportunities to contribute to the wellbeing of individuals and the advancement of economies and societies, but also a variety of novel ethical, legal, social, and technological challenges. Trustworthy AI (TAI) bases on the idea that trust builds the foundation of societies, economies, and sustainable development, and that individuals, organizations, and societies will therefore only ever be able to realize the full potential of AI, if trust can be established in its development, deployment, and use. With this article we aim to introduce the concept of TAI and its five foundational principles (1) beneficence, (2) non-maleficence, (3) autonomy, (4) justice, and (5) explicability. We further draw on these five principles to develop a data-driven research framework for TAI and demonstrate its utility by delineating fruitful avenues for future research, particularly with regard to the distributed ledger technology-based realization of TAI.
Deep neural networks have demonstrated excellent performances in many real-world applications. Unfortunately, they may show Clever Hans-like behaviour (making use of confounding factors within datasets) to achieve high performance. In this work we introduce the novel learning setting of explanatory interactive learning and illustrate its benefits on a plant phenotyping research task. Explanatory interactive learning adds the scientist into the training loop, who interactively revises the original model by providing feedback on its explanations. Our experimental results demonstrate that explanatory interactive learning can help to avoid Clever Hans moments in machine learning and encourages (or discourages, if appropriate) trust in the underlying model.
Deep learning and other black-box models are becoming more and more popular today. Despite their high performance, they may not be accepted ethically or legally because of their lack of explainability. This paper presents the increasing number of legal requirements on machine learning model interpretability and explainability in the context of private and public decision making. It then explains how those legal requirements can be implemented into machine-learning models and concludes with a call for more inter-disciplinary research on explainability.
Background
Risk scores can be useful in clinical risk stratification and accurate allocations of medical resources, helping health providers improve patient care. Point-based scores are more understandable and explainable than other complex models and are now widely used in clinical decision making. However, the development of the risk scoring model is nontrivial and has not yet been systematically presented, with few studies investigating methods of clinical score generation using electronic health records.
Objective
This study aims to propose AutoScore, a machine learning–based automatic clinical score generator consisting of 6 modules for developing interpretable point-based scores. Future users can employ the AutoScore framework to create clinical scores effortlessly in various clinical applications.
Methods
We proposed the AutoScore framework comprising 6 modules that included variable ranking, variable transformation, score derivation, model selection, score fine-tuning, and model evaluation. To demonstrate the performance of AutoScore, we used data from the Beth Israel Deaconess Medical Center to build a scoring model for mortality prediction and then compared the data with other baseline models using the receiver operating characteristic analysis. A software package in R 3.5.3 (R Foundation) was also developed to demonstrate the implementation of AutoScore.
Results
Implemented on the data set with 44,918 individual admission episodes of intensive care, the AutoScore-created scoring models performed comparably well as other standard methods (ie, logistic regression, stepwise regression, least absolute shrinkage and selection operator, and random forest) in terms of predictive accuracy and model calibration but required fewer predictors and presented high interpretability and accessibility. The nine-variable, AutoScore-created, point-based scoring model achieved an area under the curve (AUC) of 0.780 (95% CI 0.764-0.798), whereas the model of logistic regression with 24 variables had an AUC of 0.778 (95% CI 0.760-0.795). Moreover, the AutoScore framework also drives the clinical research continuum and automation with its integration of all necessary modules.
Conclusions
We developed an easy-to-use, machine learning–based automatic clinical score generator, AutoScore; systematically presented its structure; and demonstrated its superiority (predictive performance and interpretability) over other conventional methods using a benchmark database. AutoScore will emerge as a potential scoring tool in various medical applications.
Artificial intelligence (AI) systems hold great promise as decision-support tools, but we must be able to identify and understand their inevitable mistakes if they are to fulfill this potential. This is particularly true in domains where the decisions are high-stakes, such as law, medicine, and the military. In this Perspective, we describe the particular challenges for AI decision support posed in military coalition operations. These include having to deal with limited, low-quality data, which inevitably compromises AI performance. We suggest that these problems can be mitigated by taking steps that allow rapid trust calibration so that decision makers understand the AI system's limitations and likely failures and can calibrate their trust in its outputs appropriately. We propose that AI services can achieve this by being both interpretable and uncertainty-aware. Creating such AI systems poses various technical and human factors challenges. We review these challenges and recommend directions for future research.
Decision-making on numerous aspects of our daily lives is being outsourced to machine-learning (ML) algorithms and artificial intelligence (AI), motivated by speed and efficiency in the decision process. ML approaches—one of the typologies of algorithms underpinning artificial intelligence—are typically developed as black boxes. The implication is that ML code scripts are rarely scrutinised; interpretability is usually sacrificed in favour of usability and effectiveness. Room for improvement in practices associated with programme development have also been flagged along other dimensions, including inter alia fairness, accuracy, accountability, and transparency. In this contribution, the production of guidelines and dedicated documents around these themes is discussed. The following applications of AI-driven decision-making are outlined: (a) risk assessment in the criminal justice system, and (b) autonomous vehicles, highlighting points of friction across ethical principles. Possible ways forward towards the implementation of governance on AI are finally examined.
A critical procedure in diagnosing atrial fibrillation is the creation of electro-anatomic activation maps. Current methods generate these mappings from interpolation using a few sparse data points recorded inside the atria; they neither include prior knowledge of the underlying physics nor uncertainty of these recordings. Here we propose a physics-informed neural network for cardiac activation mapping that accounts for the underlying wave propagation dynamics and we quantify the epistemic uncertainty associated with these predictions. These uncertainty estimates not only allow us to quantify the predictive error of the neural network, but also help to reduce it by judiciously selecting new informative measurement locations via active learning. We illustrate the potential of our approach using a synthetic benchmark problem and a personalized electrophysiology model of the left atrium. We show that our new method outperforms linear interpolation and Gaussian process regression for the benchmark problem and linear interpolation at clinical densities for the left atrium. In both cases, the active learning algorithm achieves lower error levels than random allocation. Our findings open the door toward physics-based electro-anatomic mapping with the ultimate goals to reduce procedural time and improve diagnostic predictability for patients affected by atrial fibrillation. Open source code is available at https://github.com/fsahli/EikonalNet.
We make two theoretical contributions to disentanglement learning by (a) defining precise semantics of disentangled representations, and (b) establishing robust metrics for evaluation. First, we characterize the concept "disentangled representations" used in supervised and unsupervised methods along three dimensions-informativeness, separability and interpretability-which can be expressed and quantified explicitly using information-theoretic constructs. This helps explain the behaviors of several well-known disentanglement learning models. We then propose robust metrics for measuring informativeness, separability, and interpretability. Through a comprehensive suite of experiments, we show that our metrics correctly characterize the representations learned by different methods and are consistent with qualitative (visual) results. Thus, the metrics allow disentanglement learning methods to be compared on a fair ground. We also empirically uncovered new interesting properties of VAE-based methods and interpreted them with our formulation. These findings are promising and hopefully will encourage the design of more theoretically driven models for learning disentangled representations.
Accurate and comprehensive extraction of information from high-dimensional single cell datasets necessitates faithful visualizations to assess biological populations. A state-of-the-art algorithm for non-linear dimension reduction, t-SNE, requires multiple heuristics and fails to produce clear representations of datasets when millions of cells are projected. We develop opt-SNE, an automated toolkit for t-SNE parameter selection that utilizes Kullback-Leibler divergence evaluation in real time to tailor the early exaggeration and overall number of gradient descent iterations in a dataset-specific manner. The precise calibration of early exaggeration together with opt-SNE adjustment of gradient descent learning rate dramatically improves computation time and enables high-quality visualization of large cytometry and transcriptomics datasets, overcoming limitations of analysis tools with hard-coded parameters that often produce poorly resolved or misleading maps of fluorescent and mass cytometry data. In summary, opt-SNE enables superior data resolution in t-SNE space and thereby more accurate data interpretation. Visualisation tools that use dimensionality reduction, such as t-SNE, provide poor visualisation on large data sets of millions of observations. Here the authors present opt-SNE, that automatically finds data set-tailored parameters for t-SNE to optimise visualisation and improve analysis.
As a subfield of machine learning, reinforcement learning (RL) aims at optimizing decision making by using interaction samples of an agent with its environment and the potentially delayed feedbacks. In contrast to traditional supervised learning that typically relies on one-shot, exhaustive, and supervised reward signals, RL tackles sequential decision-making problems with sampled, evaluative, and delayed feedbacks simultaneously. Such a distinctive feature makes RL techniques a suitable candidate for developing powerful solutions in various healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged period with delayed feedbacks. By first briefly examining theoretical foundations and key methods in RL research, this survey provides an extensive overview of RL applications in a variety of healthcare domains, ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis, and many other control or scheduling problems that have infiltrated every aspect of the healthcare system. In addition, we discuss the challenges and open issues in the current research and highlight some potential solutions and directions for future research.
This article does not describe a working system. Instead, it presents a single idea about representation that allows advances made by several different groups to be combined into an imaginary system called GLOM. The advances include transformers, neural fields, contrastive representation learning, distillation, and capsules. GLOM answers the question: How can a neural network with a fixed architecture parse an image into a part-whole hierarchy that has a different structure for each image? The idea is simply to use islands of identical vectors to represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve the interpretability of the representations produced by transformer-like systems when applied to vision or language.
Model reconciliation has been proposed as a way for an agent to explain its decisions to a human who may have a different understanding of the same planning problem by explaining its decisions in terms of these model differences.However, often the human's mental model (and hence the difference) is not known precisely and such explanations cannot be readily computed.In this paper, we show how the explanation generation process evolves in the presence of such model uncertainty or incompleteness by generating {\em conformant explanations} that are applicable to a set of possible models.We also show how such explanations can contain superfluous informationand how such redundancies can be reduced using conditional explanations to iterate with the human to attain common ground. Finally, we will introduce an anytime version of this approach and empirically demonstrate the trade-offs involved in the different forms of explanations in terms of the computational overhead for the agent and the communication overhead for the human.We illustrate these concepts in three well-known planning domains as well as in a demonstration on a robot involved in a typical search and reconnaissance scenario with an external human supervisor.
Machine learning models can assist with metamaterials design by approximating computationally expensive simulators or solving inverse design problems. However, past work has usually relied on black box deep neural networks, whose reasoning processes are opaque and require enormous datasets that are expensive to obtain. In this work, we develop two novel machine learning approaches to metamaterials discovery that have neither of these disadvantages. These approaches, called shape-frequency features and unit-cell templates, can discover 2D metamaterials with user-specified frequency band gaps. Our approaches provide logical rule-based conditions on metamaterial unit-cells that allow for interpretable reasoning processes, and generalize well across design spaces of different resolutions. The templates also provide design flexibility where users can almost freely design the fine resolution features of a unit-cell without affecting the user’s desired band gap.
Black-box artificial intelligence (AI) induction methods such as deep reinforcement learning (DRL) are increasingly being used to find optimal policies for a given control task. Although policies represented using a black-box AI are capable of efficiently executing the underlying control task and achieving optimal closed-loop performance-controlling the agent from the initial time step until the successful termination of an episode, the developed control rules are often complex and neither interpretable nor explainable. In this article, we use a recently proposed nonlinear decision-tree (NLDT) approach to find a hierarchical set of control rules in an attempt to maximize the open-loop performance for approximating and explaining the pretrained black-box DRL (oracle) agent using the labeled state-action dataset. Recent advances in nonlinear optimization approaches using evolutionary computation facilitate finding a hierarchical set of nonlinear control rules as a function of state variables using a computationally fast bilevel optimization procedure at each node of the proposed NLDT. In addition, we propose a reoptimization procedure for enhancing the closed-loop performance of an already derived NLDT. We evaluate our proposed methodologies (open-and closed-loop NLDTs) on different control problems having multiple discrete actions. In all these problems, our proposed approach is able to find relatively simple and interpretable rules involving one to four nonlinear terms per rule, while simultaneously achieving on par closed-loop performance when compared to a trained black-box DRL agent. A postprocessing approach for simplifying the NLDT is also suggested. The obtained results are inspiring as they suggest the replacement of complicated black-box DRL policies involving thousands of parameters (making them noninterpretable) with relatively simple interpretable policies. The results are encouraging and motivating to pursue further applications of proposed approach in solving more complex control tasks.
Image recognition with prototypes is considered an interpretable alternative for black box deep learning models. Classification depends on the extent to which a test image “looks like” a prototype. However, perceptual similarity for humans can be different from the similarity learned by the classification model. Hence, only visualising prototypes can be insufficient for a user to understand what a prototype exactly represents, and why the model considers a prototype and an image to be similar. We address this ambiguity and argue that prototypes should be explained. We improve interpretability by automatically enhancing visual prototypes with quantitative information about visual characteristics deemed important by the classification model. Specifically, our method clarifies the meaning of a prototype by quantifying the influence of colour hue, shape, texture, contrast and saturation and can generate both global and local explanations. Because of the generality of our approach, it can improve the interpretability of any similarity-based method for prototypical image recognition. In our experiments, we apply our method to the existing Prototypical Part Network (ProtoPNet). Our analysis confirms that the global explanations are generalisable, and often correspond to the visually perceptible properties of a prototype. Our explanations are especially relevant for prototypes which might have been interpreted incorrectly otherwise. By explaining such ‘misleading’ prototypes, we improve the interpretability and simulatability of a prototype-based classification model. We also use our method to check whether visually similar prototypes have similar explanations, and are able to discover redundancy. Code is available at https://github.com/M-Nauta/Explaining_Prototypes.
2018 Curran Associates Inc.All rights reserved. While deep reinforcement learning has successfully solved many challenging control tasks, its real-world applicability has been limited by the inability to ensure the safety of learned policies. We propose an approach to verifiable reinforcement learning by training decision tree policies, which can represent complex policies (since they are nonparametric), yet can be efficiently verified using existing techniques (since they are highly structured). The challenge is that decision tree policies are difficult to train. We propose VIPER, an algorithm that combines ideas from model compression and imitation learning to learn decision tree policies guided by a DNN policy (called the oracle) and its Q-function, and show that it substantially outperforms two baselines. We use VIPER to (i) learn a provably robust decision tree policy for a variant of Atari Pong with a symbolic state space, (ii) learn a decision tree policy for a toy game based on Pong that provably never loses, and (iii) learn a provably stable decision tree policy for cart-pole. In each case, the decision tree policy achieves performance equal to that of the original DNN policy.
Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy. However, important variables for one well-performing model (for example, a linear model f (x) = x T β with a fixed coefficient vector β) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.
Numerical methods such as finite element have been flourishing in the past decades for modeling solid mechanics problems via solving governing partial differential equations (PDEs). A salient aspect that distinguishes these numerical methods is how they approximate the physical fields of interest. Physics-informed deep learning (PIDL) is a novel approach developed in recent years for modeling PDE solutions and shows promise to solve computational mechanics problems without using any labeled data (e.g., measurement data is unavailable). The philosophy behind it is to approximate the quantity of interest (e.g., PDE solution variables) by a deep neural network (DNN) and embed the physical law to regularize the network. To this end, training the network is equivalent to minimization of a well-designed loss function that contains the residuals of the governing PDEs as well as initial/boundary conditions (I/BCs). In this paper, we present a physics-informed neural network (PINN) with mixed-variable output to model elastodynamics problems without resort to the labeled data, in which the I/BCs are forcibly imposed. In particular, both the displacement and stress components are taken as the DNN output, inspired by the hybrid finite-element analysis, which largely improves the accuracy and the trainability of the network. Since the conventional PINN framework augments all the residual loss components in a soft manner with Lagrange multipliers, the weakly imposed I/BCs may not be well satisfied especially when complex I/BCs are present. To overcome this issue, a composite scheme of DNNs is established based on multiple single DNNs such that the I/BCs can be satisfied forcibly in a forcible manner. The proposed PINN framework is demonstrated on several numerical elasticity examples with different I/BCs, including both static and dynamic problems as well as wave propagation in truncated domains. Results show the promise of PINN in the context of computational mechanics applications.
As machine learning is increasingly used to help make decisions, there is a demand for these decisions to be explainable. Arguably, the most explainable machine learning models use decision rules. This paper focuses on decision sets, a type of model with unordered rules, which explains each prediction with a single rule. In order to be easy for humans to understand, these rules must be concise. Earlier work on generating optimal decision sets first minimizes the number of rules, and then minimizes the number of literals, but the resulting rules can often be very large. Here we consider a better measure, namely the total size of the decision set in terms of literals. So we are not driven to a small set of rules which require a large number of literals. We provide the first approach to determine minimum-size decision sets that achieve minimum empirical risk and then investigate sparse alternatives where we trade accuracy for size. By finding optimal solutions we show we can build decision set classifiers that are almost as accurate as the best heuristic methods, but far more concise, and hence more explainable.
In this paper, we propose an elegant solution that is directly addressing the bottlenecks of the traditional deep learning approaches and offers an explainable internal architecture that can outperform the existing methods, requires very little computational resources (no need for GPUs) and short training times (in the order of seconds). The proposed approach, xDNN is using prototypes. Prototypes are actual training data samples (images), which are local peaks of the empirical data distribution called typicality as well as of the data density. This generative model is identified in a closed form and equates to the pdf but is derived automatically and entirely from the training data with no user-or problem-specific thresholds, parameters or intervention. The proposed xDNN offers a new deep learning architecture that combines reasoning and learning in a synergy. It is non-iterative and non-parametric, which explains its efficiency in terms of time and computational resources. From the user perspective, the proposed approach is clearly understandable to human users. We tested it on challenging problems as the classification of different lighting conditions for driving scenes (iROADS), object detection (Caltech-256, and Caltech-101), and SARS-CoV-2 identification via computed tomography scan (COVID CT-scans dataset). xDNN outperforms the other methods including deep learning in terms of accuracy, time to train and offers an explainable classifier.
Recently, several exact methods to compute decision trees have been introduced. On the one hand, these approaches can find optimal trees for various objective functions including total size, depth or accuracy on the training set and therefore. On the other hand, these methods are not yet widely used in practice and classic heuristics are often still the methods of choice. In this paper we show how the SAT model proposed by [Narodytska et.al 2018] can be lifted to a MaxSAT approach, making it much more practically relevant. In particular, it scales to much larger data sets; the objective function can easily be adapted to take into account combinations of size, depth and accuracy on the training set; and the fine-grained control of the objective function it offers makes it particularly well suited for boosting. Our experiments show promising results. In particular, we show that the prediction quality of our approach often exceeds state of the art heuristics. We also show that the MaxSAT formulation is well adapted for boosting using the well-known AdaBoost Algorithm.
Several recent publications have studied the use of Mixed Integer Programming (MIP) for finding an optimal decision tree, that is, the best decision tree under formal requirements on accuracy, fairness or interpretability of the predictive model. These publications used MIP to deal with the hard computational challenge of finding such trees. In this paper, we introduce a new efficient algorithm, DL8.5, for finding optimal decision trees, based on the use of itemset mining techniques. We show that this new approach outperforms earlier approaches with several orders of magnitude, for both numerical and discrete data, and is generic as well. The key idea underlying this new approach is the use of a cache of itemsets in combination with branch-and-bound search; this new type of cache also stores results for parts of the search space that have been traversed partially.