Abstract:In humans and animals, curriculum learning -- presenting data in a curated order - is critical to rapid learning and effective pedagogy. Yet in machine learning, curricula are not widely used and empirically often yield only moderate benefits. This stark difference in the importance of curriculum raises a fundamental theoretical question: when and why does curriculum learning help? In this work, we analyse a prototypical neural network model of curriculum learning in the high-dimensional limit, employing statistical physics methods. Curricula could in principle change both the learning speed and asymptotic performance of a model. To study the former, we provide an exact description of the online learning setting, confirming the long-standing experimental observation that curricula can modestly speed up learning. To study the latter, we derive performance in a batch learning setting, in which a network trains to convergence in successive phases of learning on dataset slices of varying difficulty. With standard training losses, curriculum does not provide generalisation benefit, in line with empirical observations. However, we show that by connecting different learning phases through simple Gaussian priors, curriculum can yield a large improvement in test performance. Taken together, our reduced analytical descriptions help reconcile apparently conflicting empirical results and trace regimes where curriculum learning yields the largest gains. More broadly, our results suggest that fully exploiting a curriculum may require explicit changes to the loss function at curriculum boundaries.

The Acquisition of Physical Knowledge in Generative Neural Networks

Stochastic Gradient Descent Captures How Children Learn About Physics

Emulating Human Developmental Stages with Bayesian Neural Networks

Learning physical parameters from dynamic scenes

Intuitive physics learning in a deep-learning model inspired by developmental psychology

Curiosity-driven Intuitive Physics Learning

Mapping the Learning Curves of Deep Learning Networks

Generative Models: What Do They Know? Do They Know Things? Let's Find Out!

Probing Physics Knowledge Using Tools from Developmental Psychology

Learning Physical Intuition of Block Towers by Example

An Analytical Theory of Curriculum Learning in Teacher-Student Networks

Physics-guided generative adversarial network to learn physical models

Learning high-level visual representations from a child's perspective without strong inductive biases

Language acquisition: do children and language models follow similar learning stages?

Intrinsic Motivation Driven Intuitive Physics Learning using Deep Reinforcement Learning with Intrinsic Reward Normalization

Generative learning for nonlinear dynamics

Quick and Accurate Affordance Learning

Generative Modeling of Neural Dynamics via Latent Stochastic Differential Equations

Proximodistal Exploration in Motor Learning as an Emergent Property of Optimization

Physical Neural Networks with Self-Learning Capabilities