Abstract:Large language models can absorb a massive amount of knowledge through pretraining, but pretraining is inefficient for acquiring long-tailed or specialized facts. Therefore, fine-tuning on specialized or new knowledge that reflects changes in the world has become popular, though it risks disrupting the model's original capabilities. We study this fragility in the context of continual memorization, where the model is trained on a small set of long-tail factoids (factual associations) and must retain these factoids after multiple stages of subsequent training on other datasets. Through extensive experiments, we show that LLMs suffer from forgetting across a wide range of subsequent tasks, and simple replay techniques do not fully prevent forgetting, especially when the factoid datasets are trained in the later stages. We posit that there are two ways to alleviate forgetting: 1) protect the memorization process as the model learns the factoids, or 2) reduce interference from training in later stages. With this insight, we develop an effective mitigation strategy: REMIX (Random and Generic Data Mixing). REMIX prevents forgetting by mixing generic data sampled from pretraining corpora or even randomly generated word sequences during each stage, despite being unrelated to the memorized factoids in the first stage. REMIX can recover performance from severe forgetting, often outperforming replay-based methods that have access to the factoids from the first stage. We then analyze how REMIX alters the learning process and find that successful forgetting prevention is associated with a pattern: the model stores factoids in earlier layers than usual and diversifies the set of layers that store these factoids. The efficacy of REMIX invites further investigation into the underlying dynamics of memorization and forgetting, opening exciting possibilities for future research.

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon

SoK: Memorization in General-Purpose Large Language Models

A Multi-Perspective Analysis of Memorization in Large Language Models

Detecting Memorization in Large Language Models

Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

Emergent and Predictable Memorization in Large Language Models

Mitigating Memorization In Language Models

Beyond Memorization: The Challenge of Random Memory Access in Language Models

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Undesirable Memorization in Large Language Models: A Survey

ROME: Memorization Insights from Text, Probability and Hidden State in Large Language Models

Unlocking Memorization in Large Language Models with Dynamic Soft Prompting

Demystifying Verbatim Memorization in Large Language Models

Memorization Over Reasoning? Exposing and Mitigating Verbatim Memorization in Large Language Models' Character Understanding Evaluation

Rethinking LLM Memorization through the Lens of Adversarial Compression

Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models

Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization

Continual Memorization of Factoids in Large Language Models

Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs

Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs

Localizing Paragraph Memorization in Language Models