Abstract:With the growing complexity and capability of large language models, a need to understand model reasoning has emerged, often motivated by an underlying goal of controlling and aligning models. While numerous interpretability and steering methods have been proposed as solutions, they are typically designed either for understanding or for control, seldom addressing both, with the connection between interpretation and control more broadly remaining tenuous. Additionally, the lack of standardized applications, motivations, and evaluation metrics makes it difficult to assess these methods' practical utility and efficacy. To address this, we propose intervention as a fundamental goal of interpretability and introduce success criteria to evaluate how well methods are able to control model behavior through interventions. We unify and extend four popular interpretability methods--sparse autoencoders, logit lens, tuned lens, and probing--into an abstract encoder-decoder framework. This framework maps intermediate latent representations to human-interpretable feature spaces, enabling interventions on these interpretable features, which can then be mapped back to latent representations to control model outputs. We introduce two new evaluation metrics: intervention success rate and the coherence-intervention tradeoff, designed to measure the accuracy of explanations and their utility in controlling model behavior. Our findings reveal that (1) although current methods allow for intervention, they are inconsistent across models and features, (2) lens-based methods outperform others in achieving simple, concrete interventions, and (3) interventions often compromise model performance and coherence, underperforming simpler alternatives, such as prompting, for steering model behavior and highlighting a critical shortcoming of current interpretability approaches in real-world applications requiring control.

Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control

Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning

Towards Unifying Interpretability and Control: Evaluation via Intervention

Scaling and evaluating sparse autoencoders

Contrastive Sparse Autoencoders for Interpreting Planning of Chess-Playing Agents

The Interpretable Dictionary in Sparse Coding

Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning

Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models

A Survey of the Interpretability Aspect of Deep Learning Models

Improving Dictionary Learning with Gated Sparse Autoencoders

Decomposing The Dark Matter of Sparse Autoencoders

A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

Efficient Dictionary Learning with Switch Sparse Autoencoders

Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models

Disentangling Dense Embeddings with Sparse Autoencoders

Dictionary Evaluation and Optimization for Sparse Coding Based Speech Processing

Interpreting Attention Layer Outputs with Sparse Autoencoders

Evaluation of Dictionary for Sparse Coding in Speech Processing.

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Exploring the Latent Space of Autoencoders with Interventional Assays

SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders