Abstract:Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the reliability of their output and the privacy of their training data. In order to cleanly measure and disentangle memorisation from other phenomena (e.g. in-context learning), we create an experimental framework that is based on repeatedly exposing LLMs to random strings. Our framework allows us to better understand the dynamics, i.e., the behaviour of the model, when repeatedly exposing it to random strings. Using our framework, we make several striking observations: (a) we find consistent phases of the dynamics across families of models (Pythia, Phi and Llama2), (b) we identify factors that make some strings easier to memorise than others, and (c) we identify the role of local prefixes and global context in memorisation. We also show that sequential exposition to different random strings has a significant effect on memorisation. Our results, often surprising, have significant downstream implications in the study and usage of LLMs.

What problem does this paper attempt to address?

The paper aims to explore the ability of large language models (LLMs) to memorize data during training and its implications. Specifically, the authors created an experimental framework to study the phenomenon of memory by repeatedly exposing LLMs to random strings. This framework allows researchers to better understand memory dynamics and distinguish memory from other phenomena (such as contextual learning). The main findings include: 1. **Memory Dynamics**: The study found that all model families (Pythia, Phi, and Llama2) exhibit two consistent phases during the memory process—guessing phase and memorization phase. In the guessing phase, the model learns the probability distribution of symbols in the string; in the memorization phase, the model recalls the next symbol based on the prefix memory. 2. **Factors Affecting Memory**: Some strings are easier to memorize than others. Strings with lower entropy perform better in the guessing phase, but in the subsequent memorization phase, strings with higher entropy are remembered more quickly. 3. **Role of Global Context and Local Prefix**: Local prefixes are crucial to the memory process, but prefixes alone are not sufficient to fully recall symbols. The global context, i.e., the probability distribution of symbols in the string, significantly improves the accuracy of recalling the next symbol. 4. **Impact of Continuous Exposure to Different Random Strings**: When the model is continuously exposed to different random strings, its memory behavior changes significantly. The model forgets old strings, but as the number of exposed strings increases, the forgetting rate slows down, and the memorization rate of new strings accelerates. These results have important implications for the research and application of LLMs, especially in assessing privacy risks and understanding the memory capabilities of different types of training data. Additionally, the study reveals new threats in the memory process and provides a new theoretical foundation for future research.

Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

SoK: Memorization in General-Purpose Large Language Models

Detecting Memorization in Large Language Models

Undesirable Memorization in Large Language Models: A Survey

Emergent and Predictable Memorization in Large Language Models

Demystifying Verbatim Memorization in Large Language Models

A Multi-Perspective Analysis of Memorization in Large Language Models

Quantifying and Analyzing Entity-level Memorization in Large Language Models

MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory

Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization

Rethinking LLM Memorization through the Lens of Adversarial Compression

Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs

Schrodinger's Memory: Large Language Models

Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs

Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models

Measuring memorization through probabilistic discoverable extraction

On Memorization of Large Language Models in Logical Reasoning

Mitigating Memorization In Language Models

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon

Do LLMs Dream of Ontologies?

An evaluation on large language model outputs: Discourse and memorization