Abstract:Research on Explainable Artificial Intelligence has recently started exploring the idea of producing explanations that, rather than being expressed in terms of low-level features, are encoded in terms of interpretable concepts learned from data. How to reliably acquire such concepts is, however, still fundamentally unclear. An agreed-upon notion of concept interpretability is missing, with the result that concepts used by both post hoc explainers and concept-based neural networks are acquired through a variety of mutually incompatible strategies. Critically, most of these neglect the human side of the problem: a representation is understandable only insofar as it can be understood by the human at the receiving end. The key challenge in human-interpretable representation learning (hrl) is how to model and operationalize this human element. In this work, we propose a mathematical framework for acquiring interpretable representations suitable for both post hoc explainers and concept-based neural networks. Our formalization of hrl builds on recent advances in causal representation learning and explicitly models a human stakeholder as an external observer. This allows us derive a principled notion of alignment between the machine’s representation and the vocabulary of concepts understood by the human. In doing so, we link alignment and interpretability through a simple and intuitive name transfer game, and clarify the relationship between alignment and a well-known property of representations, namely disentanglement. We also show that alignment is linked to the issue of undesirable correlations among concepts, also known as concept leakage, and to content-style separation, all through a general information-theoretic reformulation of these properties. Our conceptualization aims to bridge the gap between the human and algorithmic sides of interpretability and establish a stepping stone for new research on human-interpretable representations.

An Interpretability Illusion for BERT

Understanding Neural Networks through Representation Erasure.

Interpretability Illusions in the Generalization of Simplified Models

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

Representations as Language: An Information-Theoretic Framework for Interpretability

Explaining Language Models' Predictions with High-Impact Concepts

Explaining Explainability: Understanding Concept Activation Vectors

Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars

BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT

Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability

Abstraction via exemplars? A representational case study on lexical category inference in BERT

Intepreting & Improving Pretrained Language Models: A Probabilistic Conceptual Approach

How is BERT surprised? Layerwise detection of linguistic anomalies

Meaningful Models: Utilizing Conceptual Structure to Improve Machine Learning Interpretability

Variational Language Concepts for Interpreting Foundation Language Models

What is Interpretability?

The mythos of model interpretability

Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability

Interpretability Is in the Mind of the Beholder: A Causal Framework for Human-Interpretable Representation Learning

Seeing in Words: Learning to Classify through Language Bottlenecks

On the Lack of Robust Interpretability of Neural Text Classifiers