Abstract:When language models (LMs) are trained to forget (or "unlearn'') a skill, how precisely does their behavior change? We study the behavior of transformer LMs in which tasks have been forgotten via fine-tuning on randomized labels. Such LMs learn to generate near-random predictions for individual examples in the "training'' set used for forgetting. Across tasks, however, LMs exhibit extreme variability in whether LM predictions change on examples outside the training set. In some tasks (like entailment classification), forgetting generalizes robustly, and causes models to produce uninformative predictions on new task instances; in other tasks (like physical commonsense reasoning and scientific question answering) forgetting affects only the training examples, and models continue to perform the "forgotten'' task accurately even for examples very similar to those that appeared in the training set. Dataset difficulty is not predictive of whether a behavior can be forgotten; instead, generalization in forgetting is (weakly) predicted by the confidence of LMs' initial task predictions and the variability of LM representations of training data, with low confidence and low variability both associated with greater generalization. Perhaps most surprisingly, random-label forgetting appears to be somewhat insensitive to the contents of the training set: for example, models trained on science questions with random labels continue to answer other science questions accurately, but begin to produce random labels on entailment classification tasks. Finally, we show that even generalizable forgetting is shallow: linear probes trained on LMs' representations can still perform tasks reliably after forgetting. Our results highlight the difficulty and unpredictability of performing targeted skill removal from models via fine-tuning.

Is generalization decay a fundamental law of psychology?

Generalization, similarity, and Bayesian inference

Unifying Principles of Generalization: Past, Present, and Future

The universal law of generalization holds for naturalistic stimuli.

The Universal Law of Generalization Holds for Naturalistic Stimuli

The generalized universal law of generalization

Psychology Is a Property of Persons, Not Averages or Distributions: Confronting the Group-to-Person Generalizability Problem in Experimental Psychology

On the generality of the laws of learning.

Relativity of Remembering: Why the Laws of Memory Vanished

Is Complexity an Illusion?

Inching Toward a Unified Metatheory for Psychology

Explaining generalization in deep learning: progress and fundamental limits

Generalization and Memorization: the Bias Potential Model.

Limitations of Neural Collapse for Understanding Generalization in Deep Learning

How to Understand Behavioral Patterns in Big Data: The Case of Human Collective Memory

Evidence for the size principle in semantic and perceptual domains

Generalization by design: Shortcuts to Generalization in Deep Learning

Behavioral momentum and the law of effect

Representations and generalization in artificial and brain neural networks

Unforgettable Generalization in Language Models

A roadmap to reverse engineering real-world generalization by combining naturalistic paradigms, deep sampling, and predictive computational models