Abstract:Continual learning (CL) aims to learn new tasks without forgetting previous tasks. However, existing CL methods require a large amount of raw data, which is often unavailable due to copyright considerations and privacy risks. Instead, stakeholders usually release pre-trained machine learning models as a service (MLaaS), which users can access via APIs. This paper considers two practical-yet-novel CL settings: data-efficient CL (DECL-APIs) and data-free CL (DFCL-APIs), which achieve CL from a stream of APIs with partial or no raw data. Performing CL under these two new settings faces several challenges: unavailable full raw data, unknown model parameters, heterogeneous models of arbitrary architecture and scale, and catastrophic forgetting of previous APIs. To overcome these issues, we propose a novel data-free cooperative continual distillation learning framework that distills knowledge from a stream of APIs into a CL model by generating pseudo data, just by querying APIs. Specifically, our framework includes two cooperative generators and one CL model, forming their training as an adversarial game. We first use the CL model and the current API as fixed discriminators to train generators via a derivative-free method. Generators adversarially generate hard and diverse synthetic data to maximize the response gap between the CL model and the API. Next, we train the CL model by minimizing the gap between the responses of the CL model and the black-box API on synthetic data, to transfer the API's knowledge to the CL model. Furthermore, we propose a new regularization term based on network similarity to prevent catastrophic forgetting of previous APIs.Our method performs comparably to classic CL with full raw data on the MNIST and SVHN in the DFCL-APIs setting. In the DECL-APIs setting, our method achieves 0.97x, 0.75x and 0.69x performance of classic CL on CIFAR10, CIFAR100, and MiniImageNet.

Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation

Computationally Budgeted Continual Learning: What Does Matter?

Continual Learning on a Data Diet

Label-Efficient Online Continual Object Detection in Streaming Video

EsaCL: Efficient Continual Learning of Sparse Models

CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One

Prior-Free Continual Learning with Unlabeled Data in the Wild

Label Delay in Online Continual Learning

Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning

An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning

Bio-inspired, task-free continual learning through activity regularization

Density Distribution-based Learning Framework for Addressing Online Continual Learning Challenges

Continual Learning From a Stream of APIs

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

A Comprehensive Empirical Evaluation on Online Continual Learning

Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling

Regularization-Based Efficient Continual Learning in Deep State-Space Models

TinySubNets: An efficient and low capacity continual learning strategy

Calibration of Continual Learning Models

The CLEAR Benchmark: Continual LEArning on Real-World Imagery

SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model