Machine Learning Interpretability Taxonomy

Anirban Nandi,Aditya Kumar Pal

DOI: https://doi.org/10.1007/978-1-4842-7802-4_3

2022-01-01

Abstract:The previous chapter explained why interpretability is important and the motivations behind building interpretable models. This chapter discusses the different themes under which different interpretable models can fit and where we should use specific methods. Figure 3-1 illustrates the different types.

What problem does this paper attempt to address?

Interpretability and Explainability: A Machine Learning Zoo Mini-tour

Ričards Marcinkevičs,Julia E. Vogt

DOI: https://doi.org/10.48550/arXiv.2012.01805

IF: 5.414

2020-12-03

Machine Learning

Abstract:In this review, we examine the problem of designing interpretable and explainable machine learning models. Interpretability and explainability lie at the core of many machine learning and statistical applications in medicine, economics, law, and natural sciences. Although interpretability and explainability have escaped a clear universal definition, many techniques motivated by these properties have been developed over the recent 30 years with the focus currently shifting towards deep learning methods. In this review, we emphasise the divide between interpretability and explainability and illustrate these two different research directions with concrete examples of the state-of-the-art. The review is intended for a general machine learning audience with interest in exploring the problems of interpretation and explanation beyond logistic regression or random forest variable importance. This work is not an exhaustive literature survey, but rather a primer focusing selectively on certain lines of research which the authors found interesting or informative.
Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

Ričards Marcinkevičs,Julia E. Vogt

DOI: https://doi.org/10.1002/widm.1493

2023-03-02

Abstract:Interpretability and explainability are essential principles of machine learning model and method design and development for medicine, economics, law, and natural sciences applications. Over the last 30 years, many techniques motivated by these properties have been developed. This review is intended for a general machine learning audience interested in exploring the challenges of interpretation and explanation beyond the logistic regression or random forest variable importance. We will examine inductive biases behind interpretable and explainable machine learning and illustrate them with concrete examples from the literature. Interpretability and explainability are crucial for machine learning (ML) and statistical applications in medicine, economics, law, and natural sciences and form an essential principle for ML model design and development. Although interpretability and explainability have escaped a precise and universal definition, many models and techniques motivated by these properties have been developed over the last 30 years, with the focus currently shifting toward deep learning. We will consider concrete examples of state‐of‐the‐art, including specially tailored rule‐based, sparse, and additive classification models, interpretable representation learning, and methods for explaining black‐box models post hoc. The discussion will emphasize the need for and relevance of interpretability and explainability, the divide between them, and the inductive biases behind the presented "zoo" of interpretable models and explanation methods. This article is categorized under: Fundamental Concepts of Data and Knowledge > Explainable AI Technologies > Machine Learning Commercial, Legal, and Ethical Issues > Social Considerations

computer science, artificial intelligence, theory & methods
Interpretable machine learning: Fundamental principles and 10 grand challenges

Cynthia Rudin,Chaofan Chen,Zhi Chen,Haiyang Huang,Lesia Semenova,Chudi Zhong

DOI: https://doi.org/10.1214/21-ss133

2022-01-01

Statistics Surveys

Abstract:Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the “Rashomon set” of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.

English Else
Machine Learning Interpretability: A Survey on Methods and Metrics

Diogo V. Carvalho,Eduardo M. Pereira,Jaime S. Cardoso

DOI: https://doi.org/10.3390/electronics8080832

IF: 2.9

2019-07-26

Electronics

Abstract:Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.

engineering, electrical & electronic,computer science, information systems,physics, applied
A Survey of Interpretable Machine Learning Methods

Yan Wang,Gulanbaier Tuerhong

DOI: https://doi.org/10.1109/VRHCIAI57205.2022.00047

2022-01-01

Abstract:Recently, people are increasingly pursuing more accurate machine learning systems. At the same time, the problem is that it is difficult for people to understand the internal logical structure of the system, and the machine learning model has become a black box. Lack of explanation is not only a practical problem, but also an ethical problem. At this point, understanding why a model makes a prediction is as important as the accuracy of the prediction. In order to balance the interpretability and accuracy of the model, many experts and scholars have done a lot of research on improving the interpretability of machine learning model, improved the interpretability of the model from different dimensions, and created a series of interpretable methods to help users understand the internal logic of the system and make the black box transparent. However, the current interpretability research is still in the stage of development, and there is no completely unified interpretable method at present. In this paper, firstly, the development status of machine learning and the importance of interpretable learning are discussed. Then, according to the current research status, interpretable methods are introduced from interpretable models and model independent categories, the research and application in the field of interpretability are discussed, and the current challenges and future development of interpretability are analyzed, To further promote the research and development of interpretability.
Model-Agnostic Interpretability of Machine Learning

Marco Tulio Ribeiro,Sameer Singh,Carlos Guestrin

DOI: https://doi.org/10.48550/arXiv.1606.05386

2016-06-17

Abstract:Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces. Thus, interpretability has become a vital concern in machine learning, and work in the area of interpretable models has found renewed interest. In some applications, such models are as accurate as non-interpretable ones, and thus are preferred for their transparency. Even when they are not accurate, they may still be preferred when interpretability is of paramount importance. However, restricting machine learning to interpretable models is often a severe limitation. In this paper we argue for explaining machine learning predictions using model-agnostic approaches. By treating the machine learning models as black-box functions, these approaches provide crucial flexibility in the choice of models, explanations, and representations, improving debugging, comparison, and interfaces for a variety of users and models. We also outline the main challenges for such methods, and review a recently-introduced model-agnostic explanation approach (LIME) that addresses these challenges.

Machine Learning
Explainable AI: A Review of Machine Learning Interpretability Methods

Pantelis Linardatos,Vasilis Papastefanopoulos,Sotiris Kotsiantis

DOI: https://doi.org/10.3390/e23010018

IF: 2.738

2020-12-25

Entropy

Abstract:Recent advances in artificial intelligence (AI) have led to its widespread industrial adoption, with machine learning systems demonstrating superhuman performance in a significant number of tasks. However, this surge in performance, has often been achieved through increased model complexity, turning such systems into “black box” approaches and causing uncertainty regarding the way they operate and, ultimately, the way that they come to decisions. This ambiguity has made it problematic for machine learning systems to be adopted in sensitive yet critical domains, where their value could be immense, such as healthcare. As a result, scientific interest in the field of Explainable Artificial Intelligence (XAI), a field that is concerned with the development of new methods that explain and interpret machine learning models, has been tremendously reignited over recent years. This study focuses on machine learning interpretability methods; more specifically, a literature review and taxonomy of these methods are presented, as well as links to their programming implementations, in the hope that this survey would serve as a reference point for both theorists and practitioners.

physics, multidisciplinary
Meaningful Models: Utilizing Conceptual Structure to Improve Machine Learning Interpretability

Nick Condry

DOI: https://doi.org/10.48550/arXiv.1607.00279

2016-07-01

Abstract:The last decade has seen huge progress in the development of advanced machine learning models; however, those models are powerless unless human users can interpret them. Here we show how the mind's construction of concepts and meaning can be used to create more interpretable machine learning models. By proposing a novel method of classifying concepts, in terms of 'form' and 'function', we elucidate the nature of meaning and offer proposals to improve model understandability. As machine learning begins to permeate daily life, interpretable models may serve as a bridge between domain-expert authors and non-expert users.

Machine Learning,Artificial Intelligence
A Survey of the Interpretability Aspect of Deep Learning Models

Eliot Spitzer,Rona Miles

DOI: https://doi.org/10.53759/0088/jbsha202303006

2023-01-05

Journal of Biomedical and Sustainable Healthcare Applications

Abstract:Deep neural networks have attained near-human degree of quality in images, textual, audio, and video recording categorization and predictions tasks. The networks, on the other hand, are still typically thought of as black-box functional probabilistic models that transfer an input data to a trained classifier. Integrating these systems into mission-critical activities like clinical diagnosis, scheduling, and management is the next stage in this human-machine evolutionary change, and it necessitates a degree of confidence in the technology output. Statistical measures are often employed to estimate an output's volatility. The idea of trust, on the other hand, is dependent on a human's sight into a machine's inner workings. To put it another way, the neural networks must justify its outputs in a way that is intelligible to humans, leading to new insights into its internal workings. "Interpretable deep networks" is the name we give to such networks. The concept of interpretability is not one-dimensional. Indeed, the variability of an interpretation owing to varying degrees of human comprehension necessitates the existence of a plethora of characteristics that together define interpretability. Furthermore, the model's interpretations may be expressed in terms of low-level network variables or input properties. We describe several of the variables that are helpful for model interpretability in this study, as well as previous work on those dimensions. We do a gap analysis to determine what remains to be improved to increase models interpretability as step of the procedure.
Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation

Raha Moraffah,Mansooreh Karami,Ruocheng Guo,Adrienne Raglin,Huan Liu

DOI: https://doi.org/10.1145/3400051.3400058

2020-01-01

ACM SIGKDD Explorations Newsletter

Abstract:Machine learning models have had discernible achievements in a myriad of applications. However, most of these models are black-boxes, and it is obscure how the decisions are made by them. This makes the models unreliable and untrustworthy. To provide insights into the decision making processes of these models, a variety of traditional interpretable models have been proposed. Moreover, to generate more human-friendly explanations, recent work on interpretability tries to answer questions related to causality such as "Why does this model makes such decisions?" or "Was it a specific feature that caused the decision made by the model?". In this work, models that aim to answer causal questions are referred to as causal interpretable models. The existing surveys have covered concepts and methodologies of traditional interpretability. In this work, we present a comprehensive survey on causal interpretable models from the aspects of the problems and methods. In addition, this survey provides in-depth insights into the existing evaluation metrics for measuring interpretability, which can help practitioners understand for what scenarios each evaluation metric is suitable.
Interpretability of deep learning models: A survey of results

Supriyo Chakraborty,Richard Tomsett,Ramya Raghavendra,Daniel Harborne,Moustafa Alzantot,Federico Cerutti,Mani Srivastava,Alun Preece,Simon Julier,Raghuveer M. Rao,Troy D. Kelley,Dave Braines,Murat Sensoy,Christopher J. Willis,Prudhvi Gurram

DOI: https://doi.org/10.1109/uic-atc.2017.8397411

2017-08-01

Abstract:Deep neural networks have achieved near-human accuracy levels in various types of classification and prediction tasks including images, text, speech, and video data. However, the networks continue to be treated mostly as black-box function approximators, mapping a given input to a classification output. The next step in this human-machine evolutionary process — incorporating these networks into mission critical processes such as medical diagnosis, planning and control — requires a level of trust association with the machine output. Typically, statistical metrics are used to quantify the uncertainty of an output. However, the notion of trust also depends on the visibility that a human has into the working of the machine. In other words, the neural network should provide human-understandable justifications for its output leading to insights about the inner workings. We call such models as interpretable deep networks. Interpretability is not a monolithic notion. In fact, the subjectivity of an interpretation, due to different levels of human understanding, implies that there must be a multitude of dimensions that together constitute interpretability. In addition, the interpretation itself can be provided either in terms of the low-level network parameters, or in terms of input features used by the model. In this paper, we outline some of the dimensions that are useful for model interpretability, and categorize prior work along those dimensions. In the process, we perform a gap analysis of what needs to be done to improve model interpretability.
The mythos of model interpretability

Zachary C. Lipton

DOI: https://doi.org/10.1145/3233231

IF: 22.7

2018-09-26

Communications of the ACM

Abstract:In machine learning, the concept of interpretability is both important and slippery.

computer science, theory & methods, software engineering, hardware & architecture
Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges

Christoph Molnar,Giuseppe Casalicchio,Bernd Bischl

DOI: https://doi.org/10.1007/978-3-030-65965-3_28

2020-10-19

Abstract:We present a brief history of the field of interpretable machine learning (IML), give an overview of state-of-the-art interpretation methods, and discuss challenges. Research in IML has boomed in recent years. As young as the field is, it has over 200 years old roots in regression modeling and rule-based machine learning, starting in the 1960s. Recently, many new IML methods have been proposed, many of them model-agnostic, but also interpretation techniques specific to deep learning and tree-based ensembles. IML methods either directly analyze model components, study sensitivity to input perturbations, or analyze local or global surrogate approximations of the ML model. The field approaches a state of readiness and stability, with many methods not only proposed in research, but also implemented in open-source software. But many important challenges remain for IML, such as dealing with dependent features, causal interpretation, and uncertainty estimation, which need to be resolved for its successful application to scientific problems. A further challenge is a missing rigorous definition of interpretability, which is accepted by the community. To address the challenges and advance the field, we urge to recall our roots of interpretable, data-driven modeling in statistics and (rule-based) ML, but also to consider other areas such as sensitivity analysis, causal inference, and the social sciences.

Machine Learning
Interpretability of Machine Learning: Recent Advances and Future Prospects

Lei Gao,Ling Guan

2023-05-01

Abstract:The proliferation of machine learning (ML) has drawn unprecedented interest in the study of various multimedia contents such as text, image, audio and video, among others. Consequently, understanding and learning ML-based representations have taken center stage in knowledge discovery in intelligent multimedia research and applications. Nevertheless, the black-box nature of contemporary ML, especially in deep neural networks (DNNs), has posed a primary challenge for ML-based representation learning. To address this black-box problem, the studies on interpretability of ML have attracted tremendous interests in recent years. This paper presents a survey on recent advances and future prospects on interpretability of ML, with several application examples pertinent to multimedia computing, including text-image cross-modal representation learning, face recognition, and the recognition of objects. It is evidently shown that the study of interpretability of ML promises an important research direction, one which is worth further investment in.

Multimedia,Computers and Society,Machine Learning
Measuring Interpretability for Different Types of Machine Learning Models

Qing Zhou,Fenglu Liao,Chao Mou,Ping Wang

DOI: https://doi.org/10.1007/978-3-030-04503-6_29

2018-01-01

Abstract:The interpretability of a machine learning model plays a significant role in practical applications, thus it is necessary to develop a method to compare the interpretability for different models so as to select the most appropriate one. However, model interpretability, a highly subjective concept, is difficult to be accurately measured, not to mention the interpretability comparison of different models. To this end, we develop an interpretability evaluation model to compute model interpretability and compare interpretability for different models. Specifically, first we we present a general form of model interpretability. Second, a questionnaire survey system is developed to collect information about users' understanding of a machine learning model. Next, three structure features are selected to investigate the relationship between interpretability and structural complexity. After this, an interpretability label is build based on the questionnaire survey result and a linear regression model is developed to evaluate the relationship between the structural features and model interpretability. The experiment results demonstrate that our interpretability evaluation model is valid and reliable to evaluate the interpretability of different models.
Data Visualization for Machine Learning Interpretability

Mukesh Sharma,Suresh Kumar Sharma

DOI: https://doi.org/10.61841/v23i4/400325

2019-01-01

International Journal of Psychosocial Rehabilitation

Abstract:Data visualization performs a vital function in enhancing the interpretability of machine learning fashions, addressing the "black box" nature of complex algorithms. As the system getting to know models come to be more and more state-of-the-art, know-how their selection-making methods will become greater challenging. Visualizations offer an intuitive manner to get to the bottom of the complex relationships within those models, providing insights into feature significance, model conduct, and ability biases. Techniques including partial dependence plots, LIME (Local Interpretable Model-agnostic Explanations), and SHAP (SHapley Additive exPlanations) values serve as powerful gear to visualize and interpret system getting to know predictions. These visualizations not best aid statistics scientists in debugging and refining models however additionally make contributions to building agree with and transparency, essential factors for broader attractiveness and deployment of machine learning answers in real-global programs. As the field of system learning progresses, exploring novel and effective methods to visualize model interpretability turns into vital for empowering each expert and non-professionals to recognise, consider, and correctly use the insights derived from complex device studying structures. Data visualization is a crucial element in unraveling the complex layers of devices, gaining knowledge of fashions, and improving their interpretability. As these models grow in complexity, expertise in their choice-making mechanisms turns into paramount for agree with, responsibility, and effective deployment. Visual representations function a powerful device to distill complex records into handy codecs, permitting stakeholders to recognise and scrutinize the version's conduct.Techniques which include function importance plots, partial dependence plots, and SHAP (SHapley Additive exPlanations) values provide insights into the impact of person features on model predictions. Feature significance plots highlight the significance of each input variable, assisting in the identity of influential elements. Partial dependence plots showcase the relationship between a specific characteristic and the version's output at the same time as keeping different variables steady, offering a nuanced know-how of their impact.SHAP values, alternatively, provide a greater holistic view by means of assigning a contribution score to every feature for every prediction, revealing the collective have an effect on of capabilities at the model's choice. These visualizations allow stakeholders to grasp not most effective which functions are critical but additionally how they have interaction, fostering a more nuanced understanding of the model's choice common sense.Moreover, confusion matrices, ROC curves, and precision-remember curves are quintessential tools for comparing version performance. These visualizations facilitate a comprehensive evaluation of class fashions by illustrating real positives, actual negatives, false positives, and false negatives. ROC curves graphically constitute the change-off among sensitivity and specificity, assisting within the selection of suitable.
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

Xuhong Li,Haoyi Xiong,Xingjian Li,Xuanyu Wu,Xiao Zhang,Ji Liu,Jiang Bian,Dejing Dou

DOI: https://doi.org/10.1007/s10115-022-01756-8

IF: 2.7

2022-09-15

Knowledge and Information Systems

Abstract:Deep neural networks have been well-known for their superb handling of various machine learning and artificial intelligence tasks. However, due to their over-parameterized black-box nature, it is often difficult to understand the prediction results of deep models. In recent years, many interpretation tools have been proposed to explain or reveal how deep models make decisions. In this paper, we review this line of research and try to make a comprehensive survey. Specifically, we first introduce and clarify two basic concepts—interpretations and interpretability—that people usually get confused about. To address the research efforts in interpretations, we elaborate the designs of a number of interpretation algorithms, from different perspectives, by proposing a new taxonomy. Then, to understand the interpretation results, we also survey the performance metrics for evaluating interpretation algorithms. Further, we summarize the current works in evaluating models' interpretability using "trustworthy" interpretation algorithms. Finally, we review and discuss the connections between deep models' interpretations and other factors, such as adversarial robustness and learning from interpretations, and we introduce several open-source libraries for interpretation algorithms and evaluation approaches.

computer science, information systems, artificial intelligence
Towards A Rigorous Science of Interpretable Machine Learning

Finale Doshi-Velez,Been Kim

2017-03-03

Abstract:As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.

Machine Learning,Artificial Intelligence
Designing Inherently Interpretable Machine Learning Models

Agus Sudjianto,Aijun Zhang

DOI: https://doi.org/10.48550/arXiv.2111.01743

IF: 5.414

2021-11-02

Machine Learning

Abstract:Interpretable machine learning (IML) becomes increasingly important in highly regulated industry sectors related to the health and safety or fundamental rights of human beings. In general, the inherently IML models should be adopted because of their transparency and explainability, while black-box models with model-agnostic explainability can be more difficult to defend under regulatory scrutiny. For assessing inherent interpretability of a machine learning model, we propose a qualitative template based on feature effects and model architecture constraints. It provides the design principles for high-performance IML model development, with examples given by reviewing our recent works on ExNN, GAMI-Net, SIMTree, and the Aletheia toolkit for local linear interpretability of deep ReLU networks. We further demonstrate how to design an interpretable ReLU DNN model with evaluation of conceptual soundness for a real case study of predicting credit default in home lending. We hope that this work will provide a practical guide of developing inherently IML models in high risk applications in banking industry, as well as other sectors.

Machine Learning Interpretability Taxonomy

Interpretability and Explainability: A Machine Learning Zoo Mini-tour

Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

Interpretable machine learning: Fundamental principles and 10 grand challenges

Machine Learning Interpretability: A Survey on Methods and Metrics

A Survey of Interpretable Machine Learning Methods

Model-Agnostic Interpretability of Machine Learning

Explainable AI: A Review of Machine Learning Interpretability Methods

Meaningful Models: Utilizing Conceptual Structure to Improve Machine Learning Interpretability

A Survey of the Interpretability Aspect of Deep Learning Models

Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation

Interpretability of deep learning models: A survey of results

The mythos of model interpretability

Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges

Interpretability of Machine Learning: Recent Advances and Future Prospects

Measuring Interpretability for Different Types of Machine Learning Models

Data Visualization for Machine Learning Interpretability

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

Towards A Rigorous Science of Interpretable Machine Learning

Designing Inherently Interpretable Machine Learning Models