Abstract:Artificial neural networks have long been understood as "black boxes": though we know their computation graphs and learned parameters, the knowledge encoded by these weights and functions they perform are not inherently interpretable. As such, from the early days of deep learning, there have been efforts to explain these models' behavior and understand them internally; and recently, mechanistic interpretability (MI) has emerged as a distinct research area studying the features and implicit algorithms learned by foundation models such as large language models. In this work, we aim to ground MI in the context of cognitive science, which has long struggled with analogous questions in studying and explaining the behavior of "black box" intelligent systems like the human brain. We leverage several important ideas and developments in the history of cognitive science to disentangle divergent objectives in MI and indicate a clear path forward. First, we argue that current methods are ripe to facilitate a transition in deep learning interpretation echoing the "cognitive revolution" in 20th-century psychology that shifted the study of human psychology from pure behaviorism toward mental representations and processing. Second, we propose a taxonomy mirroring key parallels in computational neuroscience to describe two broad categories of MI research, semantic interpretation (what latent representations are learned and used) and algorithmic interpretation (what operations are performed over representations) to elucidate their divergent goals and objects of study. Finally, we elaborate the parallels and distinctions between various approaches in both categories, analyze the respective strengths and weaknesses of representative works, clarify underlying assumptions, outline key challenges, and discuss the possibility of unifying these modes of interpretation under a common framework.

From Neurons to Neutrons: A Case Study in Interpretability

The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms

A Survey of the Interpretability Aspect of Deep Learning Models

Interpretability of Neural Network With Physiological Mechanisms

What is Interpretability?

Interpretability of deep learning models: A survey of results

Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience

Interpretability for Reliable, Efficient, and Self-Cognitive DNNs: from Theories to Applications.

Challenges in Mechanistically Interpreting Model Representations

Representations as Language: An Information-Theoretic Framework for Interpretability

A Survey on Neural Network Interpretability

How to Explain Neural Networks: an Approximation Perspective

On the Interpretability of Quantum Neural Networks

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Interpretability of Machine Learning Methods Applied to Neuroimaging

InterpretCC: Intrinsic User-Centric Interpretability through Global Mixture of Experts

Unifying and Verifying Mechanistic Interpretations: A Case Study with Group Operations

An Interpretable Neural Network Model Through Piecewise Linear Approximation

Using Degeneracy in the Loss Landscape for Mechanistic Interpretability

The Interpretability of Quantum-Inspired Neural Network

Physics-Inspired Interpretability Of Machine Learning Models