Abstract:Humans and animals routinely infer relations between different items or events and generalize these relations to novel combinations of items. This allows them to respond appropriately to radically novel circumstances and is fundamental to advanced cognition. However, how learning systems (including the brain) can implement the necessary inductive biases has been unclear. Here we investigated transitive inference (TI), a classic relational task paradigm in which subjects must learn a relation ( > and > ) and generalize it to new combinations of items ( > ). Through mathematical analysis, we found that a broad range of biologically relevant learning models (e.g. gradient flow or ridge regression) perform TI successfully and recapitulate signature behavioral patterns long observed in living subjects. First, we found that models with item-wise additive representations automatically encode transitive relations. Second, for more general representations, a single scalar “conjunctivity factor” determines model behavior on TI and, further, the principle of norm minimization (a standard statistical inductive bias) enables models with fixed, partly conjunctive representations to generalize transitively. Finally, neural networks in the “rich regime,” which enables representation learning and has been found to improve generalization, unexpectedly show poor generalization and anomalous behavior. We find that such networks implement a form of norm minimization (over hidden weights) that yields a local encoding mechanism lacking transitivity. Our findings show how minimal statistical learning principles give rise to a classical relational inductive bias (transitivity), explain empirically observed behaviors, and establish a formal approach to understanding the neural basis of relational abstraction.

Toward a Mathematical Theory of Inductive Inference

Physical limits of inference

When Is Inductive Inference Possible?

A Mathematical Benchmark for Inductive Theorem Provers

Is there an Elegant Universal Theory of Prediction?

A Formal Approach to the Problem of Logical Non-Omniscience

Open Problems in Universal Induction & Intelligence

Infinity in computable probability

Going Forth and Drawing Back: an Intensional Approach in Nonmonotonic Inference.

A Theory of Bounded Inductive Rationality

Infrafiltration Theorem and Some Inductive Sequence of Models of Generalized Second-Order Dedekind Theory of Real Numbers With Exponentially Increasing Powers

A mathematical theory of relational generalization in transitive inference

Finite Information Numbers through the Inductive Combinatorial Hierarchy

Universal Algorithmic Intelligence: A mathematical top->down approach

Progress in Computer-Assisted Inductive Theorem Proving by Human-Orientedness and Descente Infinie?

On the Proof Complexity of Deep Inference

Inference Over Programs That Make Predictions

A Circuit Complexity Formulation of Algorithmic Information Theory

Automated Induction for Complex Data Structures

Identification of Probabilities

The Inductive Logic of Information Systems