Abstract:In recent years, continual learning with pre-training (CLPT) has received widespread interest, instead of its traditional focus of training from scratch. The use of strong pre-trained models (PTMs) can greatly facilitate knowledge transfer and alleviate catastrophic forgetting, but also suffers from progressive overfitting of pre-trained knowledge into specific downstream tasks. A majority of current efforts often keep the PTMs frozen and incorporate task-specific prompts to instruct representation learning, coupled with a prompt selection process for inference. However, due to the limited capacity of prompt parameters, this strategy demonstrates only sub-optimal performance in continual learning. In comparison, tuning all parameters of PTMs often provides the greatest potential for representation learning, making sequential fine-tuning (Seq FT) a fundamental baseline that has been overlooked in CLPT. To this end, we present an in-depth analysis of the progressive overfitting problem from the lens of Seq FT. Considering that the overly fast representation learning and the biased classification layer constitute this particular problem, we introduce the advanced Slow Learner with Classifier Alignment (SLCA++) framework to unleash the power of Seq FT, serving as a strong baseline approach for CLPT. Our approach involves a Slow Learner to selectively reduce the learning rate of backbone parameters, and a Classifier Alignment to align the disjoint classification layers in a post-hoc fashion. We further enhance the efficacy of SL with a symmetric cross-entropy loss, as well as employ a parameter-efficient strategy to implement Seq FT with SLCA++. Across a variety of continual learning scenarios on image classification benchmarks, our approach provides substantial improvements and outperforms state-of-the-art methods by a large margin. Code: <a class="link-external link-https" href="https://github.com/GengDavid/SLCA" rel="external noopener nofollow">this https URL</a>.

FeTT: Continual Class Incremental Learning via Feature Transformation Tuning

Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective

SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models

HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-training

A Unified Continual Learning Framework with General Parameter-Efficient Tuning

Bridging Pre-Trained Models to Continual Learning: A Hypernetwork Based Framework with Parameter-Efficient Fine-Tuning Techniques

Does Continual Learning Equally Forget All Parameters?

Alleviating Representational Shift for Continual Fine-tuning.

TARGET: Federated Class-Continual Learning Via Exemplar-Free Distillation

Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning

HFT: Half Fine-Tuning for Large Language Models

Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion

Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion

Dual Low-Rank Adaptation for Continual Learning with Pre-Trained Models

FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers

Sparse Orthogonal Parameters Tuning for Continual Learning

Beyond Fine-Tuning: Efficient and Effective Fed-Tuning for Mobile/Web Users

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

Don't Stop Learning: Towards Continual Learning for the CLIP Model

ConPET: Continual Parameter-Efficient Tuning for Large Language Models