The mythos of model interpretability

Zachary C. Lipton

DOI: https://doi.org/10.1145/3233231

IF: 22.7

2018-09-26

Communications of the ACM

Abstract:In machine learning, the concept of interpretability is both important and slippery.

computer science, theory & methods, software engineering, hardware & architecture

What problem does this paper attempt to address?

Meaningful Models: Utilizing Conceptual Structure to Improve Machine Learning Interpretability

Nick Condry

DOI: https://doi.org/10.48550/arXiv.1607.00279

2016-07-01

Abstract:The last decade has seen huge progress in the development of advanced machine learning models; however, those models are powerless unless human users can interpret them. Here we show how the mind's construction of concepts and meaning can be used to create more interpretable machine learning models. By proposing a novel method of classifying concepts, in terms of 'form' and 'function', we elucidate the nature of meaning and offer proposals to improve model understandability. As machine learning begins to permeate daily life, interpretable models may serve as a bridge between domain-expert authors and non-expert users.

Machine Learning,Artificial Intelligence
Model-Agnostic Interpretability of Machine Learning

Marco Tulio Ribeiro,Sameer Singh,Carlos Guestrin

DOI: https://doi.org/10.48550/arXiv.1606.05386

2016-06-17

Abstract:Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces. Thus, interpretability has become a vital concern in machine learning, and work in the area of interpretable models has found renewed interest. In some applications, such models are as accurate as non-interpretable ones, and thus are preferred for their transparency. Even when they are not accurate, they may still be preferred when interpretability is of paramount importance. However, restricting machine learning to interpretable models is often a severe limitation. In this paper we argue for explaining machine learning predictions using model-agnostic approaches. By treating the machine learning models as black-box functions, these approaches provide crucial flexibility in the choice of models, explanations, and representations, improving debugging, comparison, and interfaces for a variety of users and models. We also outline the main challenges for such methods, and review a recently-introduced model-agnostic explanation approach (LIME) that addresses these challenges.

Machine Learning
Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

Ričards Marcinkevičs,Julia E. Vogt

DOI: https://doi.org/10.1002/widm.1493

2023-03-02

Abstract:Interpretability and explainability are essential principles of machine learning model and method design and development for medicine, economics, law, and natural sciences applications. Over the last 30 years, many techniques motivated by these properties have been developed. This review is intended for a general machine learning audience interested in exploring the challenges of interpretation and explanation beyond the logistic regression or random forest variable importance. We will examine inductive biases behind interpretable and explainable machine learning and illustrate them with concrete examples from the literature. Interpretability and explainability are crucial for machine learning (ML) and statistical applications in medicine, economics, law, and natural sciences and form an essential principle for ML model design and development. Although interpretability and explainability have escaped a precise and universal definition, many models and techniques motivated by these properties have been developed over the last 30 years, with the focus currently shifting toward deep learning. We will consider concrete examples of state‐of‐the‐art, including specially tailored rule‐based, sparse, and additive classification models, interpretable representation learning, and methods for explaining black‐box models post hoc. The discussion will emphasize the need for and relevance of interpretability and explainability, the divide between them, and the inductive biases behind the presented "zoo" of interpretable models and explanation methods. This article is categorized under: Fundamental Concepts of Data and Knowledge > Explainable AI Technologies > Machine Learning Commercial, Legal, and Ethical Issues > Social Considerations

computer science, artificial intelligence, theory & methods
Interpretable machine learning: Fundamental principles and 10 grand challenges

Cynthia Rudin,Chaofan Chen,Zhi Chen,Haiyang Huang,Lesia Semenova,Chudi Zhong

DOI: https://doi.org/10.1214/21-ss133

2022-01-01

Statistics Surveys

Abstract:Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the “Rashomon set” of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.

English Else
Interpretability and Explainability: A Machine Learning Zoo Mini-tour

Ričards Marcinkevičs,Julia E. Vogt

DOI: https://doi.org/10.48550/arXiv.2012.01805

IF: 5.414

2020-12-03

Machine Learning

Abstract:In this review, we examine the problem of designing interpretable and explainable machine learning models. Interpretability and explainability lie at the core of many machine learning and statistical applications in medicine, economics, law, and natural sciences. Although interpretability and explainability have escaped a clear universal definition, many techniques motivated by these properties have been developed over the recent 30 years with the focus currently shifting towards deep learning methods. In this review, we emphasise the divide between interpretability and explainability and illustrate these two different research directions with concrete examples of the state-of-the-art. The review is intended for a general machine learning audience with interest in exploring the problems of interpretation and explanation beyond logistic regression or random forest variable importance. This work is not an exhaustive literature survey, but rather a primer focusing selectively on certain lines of research which the authors found interesting or informative.
A Survey of Interpretable Machine Learning Methods

Yan Wang,Gulanbaier Tuerhong

DOI: https://doi.org/10.1109/VRHCIAI57205.2022.00047

2022-01-01

Abstract:Recently, people are increasingly pursuing more accurate machine learning systems. At the same time, the problem is that it is difficult for people to understand the internal logical structure of the system, and the machine learning model has become a black box. Lack of explanation is not only a practical problem, but also an ethical problem. At this point, understanding why a model makes a prediction is as important as the accuracy of the prediction. In order to balance the interpretability and accuracy of the model, many experts and scholars have done a lot of research on improving the interpretability of machine learning model, improved the interpretability of the model from different dimensions, and created a series of interpretable methods to help users understand the internal logic of the system and make the black box transparent. However, the current interpretability research is still in the stage of development, and there is no completely unified interpretable method at present. In this paper, firstly, the development status of machine learning and the importance of interpretable learning are discussed. Then, according to the current research status, interpretable methods are introduced from interpretable models and model independent categories, the research and application in the field of interpretability are discussed, and the current challenges and future development of interpretability are analyzed, To further promote the research and development of interpretability.
Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning

Andrew Smart,Atoosa Kasirzadeh

DOI: https://doi.org/10.1007/s00146-024-02056-1

2024-09-05

Abstract:What is it to interpret the outputs of an opaque machine learning model. One approach is to develop interpretable machine learning techniques. These techniques aim to show how machine learning models function by providing either model centric local or global explanations, which can be based on mechanistic interpretations revealing the inner working mechanisms of models or nonmechanistic approximations showing input feature output data relationships. In this paper, we draw on social philosophy to argue that interpreting machine learning outputs in certain normatively salient domains could require appealing to a third type of explanation that we call sociostructural explanation. The relevance of this explanation type is motivated by the fact that machine learning models are not isolated entities but are embedded within and shaped by social structures. Sociostructural explanations aim to illustrate how social structures contribute to and partially explain the outputs of machine learning models. We demonstrate the importance of sociostructural explanations by examining a racially biased healthcare allocation algorithm. Our proposal highlights the need for transparency beyond model interpretability, understanding the outputs of machine learning systems could require a broader analysis that extends beyond the understanding of the machine learning model itself.

Machine Learning
In Defence of Post-hoc Explainability

Nick Oh

2024-12-23

Abstract:The widespread adoption of machine learning in scientific research has created a fundamental tension between model opacity and scientific understanding. Whilst some advocate for intrinsically interpretable models, we introduce Computational Interpretabilism (CI) as a philosophical framework for post-hoc interpretability in scientific AI. Drawing parallels with human expertise, where post-hoc rationalisation coexists with reliable performance, CI establishes that scientific knowledge emerges through structured model interpretation when properly bounded by empirical validation. Through mediated understanding and bounded factivity, we demonstrate how post-hoc methods achieve epistemically justified insights without requiring complete mechanical transparency, resolving tensions between model complexity and scientific comprehension.

Machine Learning,Artificial Intelligence
A Survey of the Interpretability Aspect of Deep Learning Models

Eliot Spitzer,Rona Miles

DOI: https://doi.org/10.53759/0088/jbsha202303006

2023-01-05

Journal of Biomedical and Sustainable Healthcare Applications

Abstract:Deep neural networks have attained near-human degree of quality in images, textual, audio, and video recording categorization and predictions tasks. The networks, on the other hand, are still typically thought of as black-box functional probabilistic models that transfer an input data to a trained classifier. Integrating these systems into mission-critical activities like clinical diagnosis, scheduling, and management is the next stage in this human-machine evolutionary change, and it necessitates a degree of confidence in the technology output. Statistical measures are often employed to estimate an output's volatility. The idea of trust, on the other hand, is dependent on a human's sight into a machine's inner workings. To put it another way, the neural networks must justify its outputs in a way that is intelligible to humans, leading to new insights into its internal workings. "Interpretable deep networks" is the name we give to such networks. The concept of interpretability is not one-dimensional. Indeed, the variability of an interpretation owing to varying degrees of human comprehension necessitates the existence of a plethora of characteristics that together define interpretability. Furthermore, the model's interpretations may be expressed in terms of low-level network variables or input properties. We describe several of the variables that are helpful for model interpretability in this study, as well as previous work on those dimensions. We do a gap analysis to determine what remains to be improved to increase models interpretability as step of the procedure.
On the Relationship Between Interpretability and Explainability in Machine Learning

Benjamin Leblanc,Pascal Germain

2024-04-25

Abstract:Interpretability and explainability have gained more and more attention in the field of machine learning as they are crucial when it comes to high-stakes decisions and troubleshooting. Since both provide information about predictors and their decision process, they are often seen as two independent means for one single end. This view has led to a dichotomous literature: explainability techniques designed for complex black-box models, or interpretable approaches ignoring the many explainability tools. In this position paper, we challenge the common idea that interpretability and explainability are substitutes for one another by listing their principal shortcomings and discussing how both of them mitigate the drawbacks of the other. In doing so, we call for a new perspective on interpretability and explainability, and works targeting both topics simultaneously, leveraging each of their respective assets.

Machine Learning,Artificial Intelligence
Machine Learning Interpretability Taxonomy

Anirban Nandi,Aditya Kumar Pal

DOI: https://doi.org/10.1007/978-1-4842-7802-4_3

2022-01-01

Abstract:The previous chapter explained why interpretability is important and the motivations behind building interpretable models. This chapter discusses the different themes under which different interpretable models can fit and where we should use specific methods. Figure 3-1 illustrates the different types.
Interpretability Needs a New Paradigm

Andreas Madsen,Himabindu Lakkaraju,Siva Reddy,Sarath Chandar

2024-05-09

Abstract:Interpretability is the study of explaining models in understandable terms to humans. At present, interpretability is divided into two paradigms: the intrinsic paradigm, which believes that only models designed to be explained can be explained, and the post-hoc paradigm, which believes that black-box models can be explained. At the core of this debate is how each paradigm ensures its explanations are faithful, i.e., true to the model's behavior. This is important, as false but convincing explanations lead to unsupported confidence in artificial intelligence (AI), which can be dangerous. This paper's position is that we should think about new paradigms while staying vigilant regarding faithfulness. First, by examining the history of paradigms in science, we see that paradigms are constantly evolving. Then, by examining the current paradigms, we can understand their underlying beliefs, the value they bring, and their limitations. Finally, this paper presents 3 emerging paradigms for interpretability. The first paradigm designs models such that faithfulness can be easily measured. Another optimizes models such that explanations become faithful. The last paradigm proposes to develop models that produce both a prediction and an explanation.

Machine Learning,Computation and Language,Computer Vision and Pattern Recognition
The Definitions of Interpretability and Learning of Interpretable Models

Weishen Pan,Changshui Zhang

DOI: https://doi.org/10.48550/arXiv.2105.14171

IF: 5.414

2021-05-29

Machine Learning

Abstract:As machine learning algorithms getting adopted in an ever-increasing number of applications, interpretation has emerged as a crucial desideratum. In this paper, we propose a mathematical definition for the human-interpretable model. In particular, we define interpretability between two information process systems. If a prediction model is interpretable by a human recognition system based on the above interpretability definition, the prediction model is defined as a completely human-interpretable model. We further design a practical framework to train a completely human-interpretable model by user interactions. Experiments on image datasets show the advantages of our proposed model in two aspects: 1) The completely human-interpretable model can provide an entire decision-making process that is human-understandable; 2) The completely human-interpretable model is more robust against adversarial attacks.
Interpretability Illusions in the Generalization of Simplified Models

Dan Friedman,Andrew Lampinen,Lucas Dixon,Danqi Chen,Asma Ghandeharioun

2024-06-05

Abstract:A common method to study deep learning systems is to use simplified model representations--for example, using singular value decomposition to visualize the model's hidden states in a lower dimensional space. This approach assumes that the results of these simplifications are faithful to the original model. Here, we illustrate an important caveat to this assumption: even if the simplified representations can accurately approximate the full model on the training set, they may fail to accurately capture the model's behavior out of distribution. We illustrate this by training Transformer models on controlled datasets with systematic generalization splits, including the Dyck balanced-parenthesis languages and a code completion task. We simplify these models using tools like dimensionality reduction and clustering, and then explicitly test how these simplified proxies match the behavior of the original model. We find consistent generalization gaps: cases in which the simplified proxies are more faithful to the original model on the in-distribution evaluations and less faithful on various tests of systematic generalization. This includes cases where the original model generalizes systematically but the simplified proxies fail, and cases where the simplified proxies generalize better. Together, our results raise questions about the extent to which mechanistic interpretations derived using tools like SVD can reliably predict what a model will do in novel situations.

Machine Learning,Computation and Language
Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs

Sungsoo Ray Hong,Jessica Hullman,Enrico Bertini

DOI: https://doi.org/10.1145/3392878

2020-05-28

Abstract:As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.

English Else
Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges

Christoph Molnar,Giuseppe Casalicchio,Bernd Bischl

DOI: https://doi.org/10.1007/978-3-030-65965-3_28

2020-10-19

Abstract:We present a brief history of the field of interpretable machine learning (IML), give an overview of state-of-the-art interpretation methods, and discuss challenges. Research in IML has boomed in recent years. As young as the field is, it has over 200 years old roots in regression modeling and rule-based machine learning, starting in the 1960s. Recently, many new IML methods have been proposed, many of them model-agnostic, but also interpretation techniques specific to deep learning and tree-based ensembles. IML methods either directly analyze model components, study sensitivity to input perturbations, or analyze local or global surrogate approximations of the ML model. The field approaches a state of readiness and stability, with many methods not only proposed in research, but also implemented in open-source software. But many important challenges remain for IML, such as dealing with dependent features, causal interpretation, and uncertainty estimation, which need to be resolved for its successful application to scientific problems. A further challenge is a missing rigorous definition of interpretability, which is accepted by the community. To address the challenges and advance the field, we urge to recall our roots of interpretable, data-driven modeling in statistics and (rule-based) ML, but also to consider other areas such as sensitivity analysis, causal inference, and the social sciences.

Machine Learning
Reliability and Interpretability in Science and Deep Learning

Luigi Scorzato

DOI: https://doi.org/10.1007/s11023-024-09682-0

IF: 5.339

2024-06-26

Minds and Machines

Abstract:In recent years, the question of the reliability of Machine Learning (ML) methods has acquired significant importance, and the analysis of the associated uncertainties has motivated a growing amount of research. However, most of these studies have applied standard error analysis to ML models—and in particular Deep Neural Network (DNN) models—which represent a rather significant departure from standard scientific modelling. It is therefore necessary to integrate the standard error analysis with a deeper epistemological analysis of the possible differences between DNN models and standard scientific modelling and the possible implications of these differences in the assessment of reliability. This article offers several contributions. First, it emphasises the ubiquitous role of model assumptions (both in ML and traditional science) against the illusion of theory-free science. Secondly, model assumptions are analysed from the point of view of their (epistemic) complexity, which is shown to be language-independent. It is argued that the high epistemic complexity of DNN models hinders the estimate of their reliability and also their prospect of long term progress. Some potential ways forward are suggested. Thirdly, this article identifies the close relation between a model's epistemic complexity and its interpretability, as introduced in the context of responsible AI. This clarifies in which sense—and to what extent—the lack of understanding of a model (black-box problem) impacts its interpretability in a way that is independent of individual skills. It also clarifies how interpretability is a precondition for a plausible assessment of the reliability of any model, which cannot be based on statistical analysis alone. This article focuses on the comparison between traditional scientific models and DNN models. However, Random Forest (RF) and Logistic Regression (LR) models are also briefly considered.

computer science, artificial intelligence
Opening the Black Box of Large Language Models: Two Views on Holistic Interpretability

Haiyan Zhao,Fan Yang,Himabindu Lakkaraju,Mengnan Du

DOI: https://doi.org/10.48550/arxiv.2402.10688

2024-01-01

Abstract:As large language models (LLMs) grow more powerful, concerns around potential harms like toxicity, unfairness, and hallucination threaten user trust. Ensuring beneficial alignment of LLMs with human values through model alignment is thus critical yet challenging, requiring a deeper understanding of LLM behaviors and mechanisms. We propose opening the black box of LLMs through a framework of holistic interpretability encompassing complementary bottom-up and top-down perspectives. The bottom-up view, enabled by mechanistic interpretability, focuses on component functionalities and training dynamics. The top-down view utilizes representation engineering to analyze behaviors through hidden representations. In this paper, we review the landscape around mechanistic interpretability and representation engineering, summarizing approaches, discussing limitations and applications, and outlining future challenges in using these techniques to achieve ethical, honest, and reliable reasoning aligned with human values.
Philosophy of science at sea: Clarifying the interpretability of machine learning

Claus Beisbart,Tim Räz

DOI: https://doi.org/10.1111/phc3.12830

2022-04-19

Philosophy Compass

Abstract:In computer science, there are efforts to make machine learning more interpretable or explainable, and thus to better understand the underlying models, algorithms, and their behavior. But what exactly is interpretability, and how can it be achieved? Such questions lead into philosophical waters because their answers depend on what explanation and understanding are—and thus on issues that have been central to the philosophy of science. In this paper, we review the recent philosophical literature on interpretability. We propose a systematization in terms of four tasks for philosophers: (i) clarify the notion of interpretability, (ii) explain the value of interpretability, (iii) provide frameworks to think about interpretability, and (iv) explore important features of it to adjust our expectations about it.

The mythos of model interpretability

Meaningful Models: Utilizing Conceptual Structure to Improve Machine Learning Interpretability

Model-Agnostic Interpretability of Machine Learning

Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

Interpretable machine learning: Fundamental principles and 10 grand challenges

Interpretability and Explainability: A Machine Learning Zoo Mini-tour

A Survey of Interpretable Machine Learning Methods

Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning

In Defence of Post-hoc Explainability

A Survey of the Interpretability Aspect of Deep Learning Models

On the Relationship Between Interpretability and Explainability in Machine Learning

Machine Learning Interpretability Taxonomy

Interpretability Needs a New Paradigm

The Definitions of Interpretability and Learning of Interpretable Models

Interpretability Illusions in the Generalization of Simplified Models

Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs

Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges

Reliability and Interpretability in Science and Deep Learning

Opening the Black Box of Large Language Models: Two Views on Holistic Interpretability

Philosophy of science at sea: Clarifying the interpretability of machine learning