Abstract:Continual learning (CL) is one of the most promising trends in recent machine learning research. Its goal is to go beyond classical assumptions in machine learning and develop models and learning strategies that present high robustness in dynamic environments. This goal is realized by designing strategies that simultaneously foster the incorporation of new knowledge while avoiding forgetting past knowledge. The landscape of CL research is fragmented into several learning evaluation protocols, comprising different learning tasks, datasets, and evaluation metrics. Additionally, the benchmarks adopted so far are still distant from the complexity of real-world scenarios, and are usually tailored to highlight capabilities specific to certain strategies. In such a landscape, it is hard to clearly and objectively assess models and strategies. In this work, we fill this gap for CL on image data by introducing two novel CL benchmarks that involve multiple heterogeneous tasks from six image datasets, with varying levels of complexity and quality. Our aim is to fairly evaluate current state-of-the-art CL strategies on a common ground that is closer to complex real-world scenarios. We additionally structure our benchmarks so that tasks are presented in increasing and decreasing order of complexity—according to a curriculum—in order to evaluate if current CL models are able to exploit structure across tasks. We devote particular emphasis to providing the CL community with a rigorous and reproducible evaluation protocol for measuring the ability of a model to generalize and not to forget while learning. Furthermore, we provide an extensive experimental evaluation showing that popular CL strategies, when challenged with our proposed benchmarks, yield sub-par performance, high levels of forgetting, and present a limited ability to effectively leverage curriculum task ordering. We believe that these results highlight the need for rigorous comparisons in future CL works as well as pave the way to design new CL strategies that are able to deal with more complex scenarios.

Bi-Objective Continual Learning: Learning 'new' While Consolidating 'Known'

Adaptive Progressive Continual Learning.

CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One

Self-paced Weight Consolidation for Continual Learning

Continual Learning With Unknown Task Boundary

Imbalance Mitigation for Continual Learning via Knowledge Decoupling and Dual Enhanced Contrastive Learning

Continual Learning by Using Information of Each Class Holistically.

Dynamic Consolidation for Continual Learning

CBA: Improving Online Continual Learning Via Continual Bias Adaptor

Contrastive Continual Learning with Importance Sampling and Prototype-Instance Relation Distillation

Dual-CBA: Improving Online Continual Learning via Dual Continual Bias Adaptors from a Bi-level Optimization Perspective

Don't Stop Learning: Towards Continual Learning for the CLIP Model

Modeling Inner- and Cross-Task Contrastive Relations for Continual Image Classification

From MNIST to ImageNet and back: benchmarking continual curriculum learning

Reinforced Continual Learning

Continual Learning Based on OOD Detection and Task Masking

CLUE: Consolidating Learned and Undergoing Experience in Domain-Incremental Classification

Continual Learning With Knowledge Distillation: A Survey

Online Active Continual Learning for Robotic Lifelong Object Recognition

Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion