Abstract:This paper studies the evolving domain of Continual Learning (CL) in large language models (LLMs), with a focus on developing strategies for efficient and sustainable training. Our primary emphasis is on continual domain-adaptive pretraining, a process designed to equip LLMs with the ability to integrate new information from various domains while retaining previously learned knowledge and enhancing cross-domain knowledge transfer without relying on domain-specific identification. Unlike previous studies, which mostly concentrate on a limited selection of tasks or domains and primarily aim to address the issue of forgetting, our research evaluates the adaptability and capabilities of LLMs to changing data landscapes in practical scenarios. To this end, we introduce a new benchmark designed to measure the adaptability of LLMs to these evolving data environments, offering a comprehensive framework for evaluation. We examine the impact of model size on learning efficacy and forgetting, as well as how the progression and similarity of emerging domains affect the knowledge transfer within these models. Our findings uncover several key insights: (i) when the sequence of domains shows semantic similarity, continual pretraining enables LLMs to better specialize in the current domain compared to stand-alone fine-tuning, (ii) training across a diverse range of domains enhances both backward and forward knowledge transfer, and (iii) smaller models are particularly sensitive to continual pretraining, showing the most significant rates of both forgetting and learning. We posit that our research marks a shift towards establishing a more realistic benchmark for investigating CL in LLMs, and has the potential to play a key role in guiding the direction of future research in the field.

Continual Training of Language Models for Few-Shot Learning

Continual Training of Language Models for Few-Shot Learning

Continual Post-Training of Language Models

Continual Learning for Large Language Models: A Survey

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

Continual Learning of Large Language Models: A Comprehensive Survey

Large Language Model Can Continue Evolving From Mistakes

Learning to Learn for Few-shot Continual Active Learning

Don't Stop Learning: Towards Continual Learning for the CLIP Model

Scalable Language Model with Generalized Continual Learning

EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data

Continual Local Replacement for Few-shot Image Recognition

Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale

Towards Effective and Efficient Continual Pre-training of Large Language Models

Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model

Efficient Continual Pre-training of LLMs for Low-resource Languages

Investigating Continual Pretraining in Large Language Models: Insights and Implications

Recent Advances of Foundation Language Models-based Continual Learning: A Survey

Online Continual Knowledge Learning for Language Models

Few-Shot Classification & Segmentation Using Large Language Models Agent

Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene