Abstract:Compositionality has long been considered a key explanatory property underlying human intelligence: arbitrary concepts can be composed into novel complex combinations, permitting the acquisition of an open ended, potentially infinite expressive capacity from finite learning experiences. Influential arguments have held that neural networks fail to explain this aspect of behavior, leading many to dismiss them as viable models of human cognition. Over the last decade, however, modern deep neural networks (DNNs), which share the same fundamental design principles as their predecessors, have come to dominate artificial intelligence, exhibiting the most advanced cognitive behaviors ever demonstrated in machines. In particular, large language models (LLMs), DNNs trained to predict the next word on a large corpus of text, have proven capable of sophisticated behaviors such as writing syntactically complex sentences without grammatical errors, producing cogent chains of reasoning, and even writing original computer programs -- all behaviors thought to require compositional processing. In this chapter, we survey recent empirical work from machine learning for a broad audience in philosophy, cognitive science, and neuroscience, situating recent breakthroughs within the broader context of philosophical arguments about compositionality. In particular, our review emphasizes two approaches to endowing neural networks with compositional generalization capabilities: (1) architectural inductive biases, and (2) metalearning, or learning to learn. We also present findings suggesting that LLM pretraining can be understood as a kind of metalearning, and can thereby equip DNNs with compositional generalization abilities in a similar way. We conclude by discussing the implications that these findings may have for the study of compositionality in human cognition and by suggesting avenues for future research.

Generalisation towards Combinatorial Productivity in Language Acquisition by Simple Recurrent Networks

Linguistic generalization and compositionality in modern artificial neural networks

Approaching Human Language with Complex Networks.

Towards a neural architecture of language: Deep learning versus logistics of access in neural architectures for compositional processing

Emergent Language Generalization and Acquisition Speed are not tied to Compositionality

Language Is a Complex Network

Towards a theory of how the structure of language is acquired by deep neural networks

The neural basis of combinatory syntax and semantics

Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings

Rule-Based and Word-Level Statistics-Based Processing of Language: Insights from Neuroscience

Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability

How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech

On (Emergent) Systematic Generalisation and Compositionality in Visual Referential Games with Straight-Through Gumbel-Softmax Estimator

Linguistic Structure from a Bottleneck on Sequential Information Processing

Emergence of compositional language with deep generational transmission

What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure

Defending Compositionality in Emergent Languages

From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks

Natural language instructions induce compositional generalization in networks of neurons

Language processing in real and artifical neural networks