Abstract:In humans and animals, curriculum learning -- presenting data in a curated order - is critical to rapid learning and effective pedagogy. Yet in machine learning, curricula are not widely used and empirically often yield only moderate benefits. This stark difference in the importance of curriculum raises a fundamental theoretical question: when and why does curriculum learning help? In this work, we analyse a prototypical neural network model of curriculum learning in the high-dimensional limit, employing statistical physics methods. Curricula could in principle change both the learning speed and asymptotic performance of a model. To study the former, we provide an exact description of the online learning setting, confirming the long-standing experimental observation that curricula can modestly speed up learning. To study the latter, we derive performance in a batch learning setting, in which a network trains to convergence in successive phases of learning on dataset slices of varying difficulty. With standard training losses, curriculum does not provide generalisation benefit, in line with empirical observations. However, we show that by connecting different learning phases through simple Gaussian priors, curriculum can yield a large improvement in test performance. Taken together, our reduced analytical descriptions help reconcile apparently conflicting empirical results and trace regimes where curriculum learning yields the largest gains. More broadly, our results suggest that fully exploiting a curriculum may require explicit changes to the loss function at curriculum boundaries.

An Empirical Exploration of Curriculum Learning for Neural Machine Translation

Reinforcement Learning based Curriculum Optimization for Neural Machine Translation

Self-Guided Curriculum Learning for Neural Machine Translation

Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation

Norm-Based Curriculum Learning for Neural Machine Translation

Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models

Token-wise Curriculum Learning for Neural Machine Translation

Data Selection Curriculum for Neural Machine Translation

Curriculum learning for language modeling

Dynamic Curriculum Learning for Low-Resource Neural Machine Translation

An Analytical Theory of Curriculum Learning in Teacher-Student Networks

Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks

Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translation

Massive Exploration of Neural Machine Translation Architectures

Optimizing the Training Schedule of Multilingual NMT using Reinforcement Learning

Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation

Curriculum Pre-training for End-to-End Speech Translation

Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks

Curriculum Learning for Small Code Language Models

Curriculum pre-training for stylized neural machine translation

Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation