Abstract:Compositionality has long been considered a key explanatory property underlying human intelligence: arbitrary concepts can be composed into novel complex combinations, permitting the acquisition of an open ended, potentially infinite expressive capacity from finite learning experiences. Influential arguments have held that neural networks fail to explain this aspect of behavior, leading many to dismiss them as viable models of human cognition. Over the last decade, however, modern deep neural networks (DNNs), which share the same fundamental design principles as their predecessors, have come to dominate artificial intelligence, exhibiting the most advanced cognitive behaviors ever demonstrated in machines. In particular, large language models (LLMs), DNNs trained to predict the next word on a large corpus of text, have proven capable of sophisticated behaviors such as writing syntactically complex sentences without grammatical errors, producing cogent chains of reasoning, and even writing original computer programs -- all behaviors thought to require compositional processing. In this chapter, we survey recent empirical work from machine learning for a broad audience in philosophy, cognitive science, and neuroscience, situating recent breakthroughs within the broader context of philosophical arguments about compositionality. In particular, our review emphasizes two approaches to endowing neural networks with compositional generalization capabilities: (1) architectural inductive biases, and (2) metalearning, or learning to learn. We also present findings suggesting that LLM pretraining can be understood as a kind of metalearning, and can thereby equip DNNs with compositional generalization abilities in a similar way. We conclude by discussing the implications that these findings may have for the study of compositionality in human cognition and by suggesting avenues for future research.

Compositional Generalization by Learning Analytical Expressions.

Compositional Substitutivity of Visual Reasoning for Visual Question Answering

Compositional generalization through abstract representations in human and artificial neural networks

A Study of Compositional Generalization in Neural Models

Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings

Concepts, Properties and an Approach for Compositional Generalization

A General Theory for Compositional Generalization

Learning Algebraic Recombination for Compositional Generalization

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization

Compositional learning of functions in humans and machines

Consistency of Compositional Generalization across Multiple Levels

Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language

Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Language

Towards Understanding the Relationship between In-context Learning and Compositional Generalization

When does compositional structure yield compositional generalization? A kernel theory

Learning to Embed Seen/Unseen Compositions based on Graph Networks

Compositionality as Lexical Symmetry

Learning to Infer Unseen Single-/ Multi-Attribute-Object Compositions with Graph Networks.

Compositional diversity in visual concept learning

From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks