The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms

Adam Davies,Ashkan Khakzar

2024-08-12

Abstract:Artificial neural networks have long been understood as "black boxes": though we know their computation graphs and learned parameters, the knowledge encoded by these weights and functions they perform are not inherently interpretable. As such, from the early days of deep learning, there have been efforts to explain these models' behavior and understand them internally; and recently, mechanistic interpretability (MI) has emerged as a distinct research area studying the features and implicit algorithms learned by foundation models such as large language models. In this work, we aim to ground MI in the context of cognitive science, which has long struggled with analogous questions in studying and explaining the behavior of "black box" intelligent systems like the human brain. We leverage several important ideas and developments in the history of cognitive science to disentangle divergent objectives in MI and indicate a clear path forward. First, we argue that current methods are ripe to facilitate a transition in deep learning interpretation echoing the "cognitive revolution" in 20th-century psychology that shifted the study of human psychology from pure behaviorism toward mental representations and processing. Second, we propose a taxonomy mirroring key parallels in computational neuroscience to describe two broad categories of MI research, semantic interpretation (what latent representations are learned and used) and algorithmic interpretation (what operations are performed over representations) to elucidate their divergent goals and objects of study. Finally, we elaborate the parallels and distinctions between various approaches in both categories, analyze the respective strengths and weaknesses of representative works, clarify underlying assumptions, outline key challenges, and discuss the possibility of unifying these modes of interpretation under a common framework.

Artificial Intelligence

What problem does this paper attempt to address?

This paper aims to address the interpretability issue of artificial neural networks (especially large language models) as "black box" systems. Specifically, the authors seek to combine Mechanistic Interpretability (MI) with cognitive science, drawing on important ideas and developments in cognitive science to distinguish different goals within MI and indicate the way forward. The paper proposes a classification system that divides MI research into semantic interpretability (exploring what the latent representations have learned) and algorithmic interpretability (exploring what operations are performed on these representations), thereby clarifying the different goals and research objects of these two types of research. Additionally, the paper discusses in detail the parallel challenges between the two types of methods, their respective advantages and disadvantages, and explores the possibility of unifying these two modes of interpretation within a common framework. In this way, the paper hopes to promote a cognitive revolution in the interpretability of deep learning, similar to the shift in 20th-century psychology from behaviorism to the study of mental representations and information processing.