Abstract:Unlearning in Large Language Models (LLMs) is essential for ensuring ethical and responsible AI use, especially in addressing privacy leak, bias, safety, and evolving regulations. Existing approaches to LLM unlearning often rely on retain data or a reference LLM, yet they struggle to adequately balance unlearning performance with overall model utility. This challenge arises because leveraging explicit retain data or implicit knowledge of retain data from a reference LLM to fine-tune the model tends to blur the boundaries between the forgotten and retain data, as different queries often elicit similar responses. In this work, we propose eliminating the need to retain data or the reference LLM for response calibration in LLM unlearning. Recognizing that directly applying gradient ascent on the forget data often leads to optimization instability and poor performance, our method guides the LLM on what not to respond to, and importantly, how to respond, based on the forget data. Hence, we introduce Forget data only Loss AjustmenT (FLAT), a "flat" loss adjustment approach which addresses these issues by maximizing f-divergence between the available template answer and the forget answer only w.r.t. the forget data. The variational form of the defined f-divergence theoretically provides a way of loss adjustment by assigning different importance weights for the learning w.r.t. template responses and the forgetting of responses subject to unlearning. Empirical results demonstrate that our approach not only achieves superior unlearning performance compared to existing methods but also minimizes the impact on the model's retained capabilities, ensuring high utility across diverse tasks, including copyrighted content unlearning on Harry Potter dataset and MUSE Benchmark, and entity unlearning on the TOFU dataset.

Learning to Unlearn for Robust Machine Unlearning

Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning

Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement

MUter: Machine Unlearning on Adversarially Trained Models

Machine Unlearning in Forgettability Sequence

A hybrid framework for effective and efficient machine unlearning

Towards Natural Machine Unlearning

MUNBa: Machine Unlearning via Nash Bargaining

Towards Robust and Cost-Efficient Knowledge Unlearning for Large Language Models

Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearning

Remaining-data-free Machine Unlearning by Suppressing Sample Contribution

Rethinking Machine Unlearning for Large Language Models

Deep Unlearn: Benchmarking Machine Unlearning

LLM Unlearning via Loss Adjustment with Only Forget Data

An Information Theoretic Approach to Machine Unlearning

Unlearn What You Want to Forget: Efficient Unlearning for LLMs

Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate

Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification

Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods

A Closer Look at Machine Unlearning for Large Language Models

A Duty to Forget, a Right to Be Assured? Exposing Vulnerabilities in Machine Unlearning Services