Abstract:Large language models can absorb a massive amount of knowledge through pretraining, but pretraining is inefficient for acquiring long-tailed or specialized facts. Therefore, fine-tuning on specialized or new knowledge that reflects changes in the world has become popular, though it risks disrupting the model's original capabilities. We study this fragility in the context of continual memorization, where the model is trained on a small set of long-tail factoids (factual associations) and must retain these factoids after multiple stages of subsequent training on other datasets. Through extensive experiments, we show that LLMs suffer from forgetting across a wide range of subsequent tasks, and simple replay techniques do not fully prevent forgetting, especially when the factoid datasets are trained in the later stages. We posit that there are two ways to alleviate forgetting: 1) protect the memorization process as the model learns the factoids, or 2) reduce interference from training in later stages. With this insight, we develop an effective mitigation strategy: REMIX (Random and Generic Data Mixing). REMIX prevents forgetting by mixing generic data sampled from pretraining corpora or even randomly generated word sequences during each stage, despite being unrelated to the memorized factoids in the first stage. REMIX can recover performance from severe forgetting, often outperforming replay-based methods that have access to the factoids from the first stage. We then analyze how REMIX alters the learning process and find that successful forgetting prevention is associated with a pattern: the model stores factoids in earlier layers than usual and diversifies the set of layers that store these factoids. The efficacy of REMIX invites further investigation into the underlying dynamics of memorization and forgetting, opening exciting possibilities for future research.

An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates

Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA

Scaling Laws for Forgetting When Fine-Tuning Large Language Models

Continual Forgetting for Pre-trained Vision Models

Dual Low-Rank Adaptation for Continual Learning with Pre-Trained Models

LoRA Learns Less and Forgets Less

Training Neural Networks from Scratch with Parallel Low-Rank Adapters

Demystifying Language Model Forgetting with Low-rank Example Associations

InRank: Incremental Low-Rank Learning

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models

AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

Continual Pre-Training Mitigates Forgetting in Language and Vision

Catastrophic Forgetting in the Context of Model Updates

Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models

Continual Memorization of Factoids in Large Language Models

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation

On Local Overfitting and Forgetting in Deep Neural Networks

Fortuitous Forgetting in Connectionist Networks

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models