Abstract:Humans and animals routinely infer relations between different items or events and generalize these relations to novel combinations of items. This allows them to respond appropriately to radically novel circumstances and is fundamental to advanced cognition. However, how learning systems (including the brain) can implement the necessary inductive biases has been unclear. Here we investigated transitive inference (TI), a classic relational task paradigm in which subjects must learn a relation ( > and > ) and generalize it to new combinations of items ( > ). Through mathematical analysis, we found that a broad range of biologically relevant learning models (e.g. gradient flow or ridge regression) perform TI successfully and recapitulate signature behavioral patterns long observed in living subjects. First, we found that models with item-wise additive representations automatically encode transitive relations. Second, for more general representations, a single scalar “conjunctivity factor” determines model behavior on TI and, further, the principle of norm minimization (a standard statistical inductive bias) enables models with fixed, partly conjunctive representations to generalize transitively. Finally, neural networks in the “rich regime,” which enables representation learning and has been found to improve generalization, unexpectedly show poor generalization and anomalous behavior. We find that such networks implement a form of norm minimization (over hidden weights) that yields a local encoding mechanism lacking transitivity. Our findings show how minimal statistical learning principles give rise to a classical relational inductive bias (transitivity), explain empirically observed behaviors, and establish a formal approach to understanding the neural basis of relational abstraction.

Unifying Principles of Generalization: Past, Present, and Future

A Kind of Generalized Learning Model

Generalization, similarity, and Bayesian inference

How Do We Generalize

Generalization in Neural Networks: A Broad Survey

A Short Survey of Systematic Generalization

The Evolution of Domain-General Mechanisms in Intelligence and Learning

On the Complexity of Bayesian Generalization

Computational principles of intelligence: learning and reasoning with neural networks

Model-free, Model-based, and General Intelligence

A roadmap to reverse engineering real-world generalization by combining naturalistic paradigms, deep sampling, and predictive computational models

Learning Whenever Learning is Possible: Universal Learning under General Stochastic Processes

On the generality of the laws of learning.

Improving Systematic Generalization Through Modularity and Augmentation

Representations and generalization in artificial and brain neural networks

The generalized universal law of generalization

Explaining generalization in deep learning: progress and fundamental limits

The universal law of generalization holds for naturalistic stimuli.

Aligning Generalisation Between Humans and Machines

The Neural Correlates of Similarity- and Rule-based Generalization

A mathematical theory of relational generalization in transitive inference