Abstract:With the increasing adoption of predictive models trained using machine learning across a wide range of high-stakes applications, e.g., health care, security, criminal justice, finance, and education, there is a growing need for effective techniques for explaining such models and their predictions. We aim to address this problem in settings where the predictive model is a black box; That is, we can only observe the response of the model to various inputs, but have no knowledge about the internal structure of the predictive model, its parameters, the objective function, and the algorithm used to optimize the model. We reduce the problem of interpreting a black box predictive model to that of estimating the causal effects of each of the model inputs on the model output, from observations of the model inputs and the corresponding outputs. We estimate the causal effects of model inputs on model output using variants of the Rubin Neyman potential outcomes framework for estimating causal effects from observational data. We show how the resulting causal attribution of responsibility for model output to the different model inputs can be used to interpret the predictive model and to explain its predictions. We present results of experiments that demonstrate the effectiveness of our approach to the interpretation of black box predictive models via causal attribution in the case of deep neural network models trained on one synthetic data set (where the input variables that impact the output variable are known by design) and two real-world data sets: Handwritten digit classification, and Parkinson's disease severity prediction. Because our approach does not require knowledge about the predictive model algorithm and is free of assumptions regarding the black box predictive model except that its input-output responses be observable, it can be applied, in principle, to any black box predictive model.

Trying to outrun causality with machine learning: Limitations of model explainability techniques for exploratory research

"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations

Scientific Exploration and Explainable Artificial Intelligence

On the Relationship Between Explanation and Prediction: A Causal View

Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction

Explaining Explanations in AI

Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

Understanding Disparities in Post Hoc Machine Learning Explanation

Can predictive models be used for causal inference?

Stop ordering machine learning algorithms by their explainability! A user-centered investigation of performance and explainability

Model Interpretation and Explainability: Towards Creating Transparency in Prediction Models

Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users' Information Processing

Interpretable machine learning for psychological research: Opportunities and pitfalls.

Explaining machine learning models using entropic variable projection

Interpretability and Explainability: A Machine Learning Zoo Mini-tour

Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review

Assessing the Reliability of Machine Learning Models Applied to the Mental Health Domain Using Explainable AI

Explaining the Behavior of Black-Box Prediction Algorithms with Causal Learning

Explaining Predictions from Machine Learning Models: Algorithms, Users, and Pedagogy

A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution

Explainability as statistical inference