Abstract:In recent years, continual learning with pre-training (CLPT) has received widespread interest, instead of its traditional focus of training from scratch. The use of strong pre-trained models (PTMs) can greatly facilitate knowledge transfer and alleviate catastrophic forgetting, but also suffers from progressive overfitting of pre-trained knowledge into specific downstream tasks. A majority of current efforts often keep the PTMs frozen and incorporate task-specific prompts to instruct representation learning, coupled with a prompt selection process for inference. However, due to the limited capacity of prompt parameters, this strategy demonstrates only sub-optimal performance in continual learning. In comparison, tuning all parameters of PTMs often provides the greatest potential for representation learning, making sequential fine-tuning (Seq FT) a fundamental baseline that has been overlooked in CLPT. To this end, we present an in-depth analysis of the progressive overfitting problem from the lens of Seq FT. Considering that the overly fast representation learning and the biased classification layer constitute this particular problem, we introduce the advanced Slow Learner with Classifier Alignment (SLCA++) framework to unleash the power of Seq FT, serving as a strong baseline approach for CLPT. Our approach involves a Slow Learner to selectively reduce the learning rate of backbone parameters, and a Classifier Alignment to align the disjoint classification layers in a post-hoc fashion. We further enhance the efficacy of SL with a symmetric cross-entropy loss, as well as employ a parameter-efficient strategy to implement Seq FT with SLCA++. Across a variety of continual learning scenarios on image classification benchmarks, our approach provides substantial improvements and outperforms state-of-the-art methods by a large margin. Code: <a class="link-external link-https" href="https://github.com/GengDavid/SLCA" rel="external noopener nofollow">this https URL</a>.

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Adversarial Supervised Contrastive Learning

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

Simple Flow-Based Contrastive Learning for BERT Sentence Representations

Label Anchored Contrastive Learning for Language Understanding

CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models

Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting

MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding

Generalized Supervised Contrastive Learning

Simple Contrastive Representation Adversarial Learning for NLP Tasks

CAVL: Learning Contrastive and Adaptive Representations of Vision and Language

Improved Visual Fine-tuning with Natural Language Supervision

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusion

CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuning

Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models

Enhancing Visual Continual Learning with Language-Guided Supervision

SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-training

Contrastive fine-tuning for low-resource graph-level transfer learning

Non-Contrastive Learning Meets Language-Image Pre-Training