Abstract:Explanations are hypothesized to improve human understanding of machine learning models and achieve a variety of desirable outcomes, ranging from model debugging to enhancing human decision making. However, empirical studies have found mixed and even negative results. An open question, therefore, is under what conditions explanations can improve human understanding and in what way. Using adapted causal diagrams, we provide a formal characterization of the interplay between machine explanations and human understanding, and show how human intuitions play a central role in enabling human understanding. Specifically, we identify three core concepts of interest that cover all existing quantitative measures of understanding in the context of human-AI decision making: task decision boundary, model decision boundary, and model error. Our key result is that without assumptions about task-specific intuitions, explanations may potentially improve human understanding of model decision boundary, but they cannot improve human understanding of task decision boundary or model error. To achieve complementary human-AI performance, we articulate possible ways on how explanations need to work with human intuitions. For instance, human intuitions about the relevance of features (e.g., education is more important than age in predicting a person's income) can be critical in detecting model error. We validate the importance of human intuitions in shaping the outcome of machine explanations with empirical human-subject studies. Overall, our work provides a general framework along with actionable implications for future algorithmic development and empirical experiments of machine explanations.

Personalized explanation in machine learning: A conceptualization

Providing personalized Explanations: a Conversational Approach

An Evaluation of the Human-Interpretability of Explanation

Explainability for Machine Learning Models: From Data Adaptability to User Perception

Explanation as a process: user-centric construction of multi-level and multi-modal explanations

Machine Explanations and Human Understanding

Not all users are the same: Providing personalized explanations for sequential decision making problems

Teaching Meaningful Explanations

How Personal is Machine Learning Personalization?

User-Centric Explainability in Healthcare: A Knowledge-Level Perspective of Informed Machine Learning

Is More Always Better? The Effects of Personal Characteristics and Level of Detail on the Perception of Explanations in a Recommender System

User Characteristics in Explainable AI: The Rabbit Hole of Personalization?

Do you comply with AI? -- Personalized explanations of learning algorithms and their impact on employees' compliance behavior

Explanation Ontology: A Model of Explanations for User-Centered AI

Explaining Explanations in AI

Personalizing explanations of AI-driven hints to users' cognitive abilities: an empirical evaluation

Crowdsourcing and Evaluating Concept-driven Explanations of Machine Learning Models

What Makes a Good Explanation?: A Harmonized View of Properties of Explanations

Explainable machine learning in deployment

Learning by Self-Explaining

Minimalistic Explanations: Capturing the Essence of Decisions