Abstract:Continual learning (CL) is one of the most promising trends in recent machine learning research. Its goal is to go beyond classical assumptions in machine learning and develop models and learning strategies that present high robustness in dynamic environments. The landscape of CL research is fragmented into several learning evaluation protocols, comprising different learning tasks, datasets, and evaluation metrics. Additionally, the benchmarks adopted so far are still distant from the complexity of real-world scenarios, and are usually tailored to highlight capabilities specific to certain strategies. In such a landscape, it is hard to objectively assess strategies. In this work, we fill this gap for CL on image data by introducing two novel CL benchmarks that involve multiple heterogeneous tasks from six image datasets, with varying levels of complexity and quality. Our aim is to fairly evaluate current state-of-the-art CL strategies on a common ground that is closer to complex real-world scenarios. We additionally structure our benchmarks so that tasks are presented in increasing and decreasing order of complexity -- according to a curriculum -- in order to evaluate if current CL models are able to exploit structure across tasks. We devote particular emphasis to providing the CL community with a rigorous and reproducible evaluation protocol for measuring the ability of a model to generalize and not to forget while learning. Furthermore, we provide an extensive experimental evaluation showing that popular CL strategies, when challenged with our benchmarks, yield sub-par performance, high levels of forgetting, and present a limited ability to effectively leverage curriculum task ordering. We believe that these results highlight the need for rigorous comparisons in future CL works as well as pave the way to design new CL strategies that are able to deal with more complex scenarios.

CurML: A Curriculum Machine Learning Library

A Comprehensive Survey on Curriculum Learning

Curriculum Learning: Theories, Approaches, Applications, Tools, and Future Directions in the Era of Large Language Models

A Survey on Curriculum Learning

CurBench: Curriculum Learning Benchmark

CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models

Review and Arrange: Curriculum Learning for Natural Language Understanding

Competence-based Curriculum Learning for Multilingual Machine Translation

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

Learning to Learn: How to Continuously Teach Humans and Machines

Syllabus: Portable Curricula for Reinforcement Learning Agents

CMLCompiler: A Unified Compiler for Classical Machine Learning

Multi-Modal Curriculum Learning over Graphs

Curriculum Learning with Adam: The Devil Is in the Wrong Details

From MNIST to ImageNet and Back: Benchmarking Continual Curriculum Learning

Curriculumformer: Taming Curriculum Pre-Training for Enhanced 3-D Point Cloud Understanding

Machine Learning Made Easy (MLme): a comprehensive toolkit for machine learning–driven data analysis

A Psychology-based Unified Dynamic Framework for Curriculum Learning

A Comprehensive Study of Privacy Risks in Curriculum Learning

Curriculum Learning: A Survey

CLIMB: Curriculum Learning for Infant-inspired Model Building