Abstract:Large language models (LLMs) with one or more fine-tuning phases have become a necessary step to unlock various capabilities, enabling LLMs to follow natural language instructions or align with human preferences. However, it carries the risk of catastrophic forgetting during sequential training, the parametric knowledge or the ability learned in previous stages may be overwhelmed by incoming training data. In this paper, we find that by regularly resetting partial parameters, LLMs can restore some of the original knowledge. Inspired by this, we introduce Half Fine-Tuning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues, where half of the parameters are selected to learn new tasks while the other half are frozen to remain previous knowledge. We provide a feasibility analysis from the perspective of optimization and interpret the parameter selection operation as a regularization term. Without changing the model architecture, HFT could be seamlessly integrated into existing fine-tuning frameworks. Extensive experiments and analysis on supervised fine-tuning, direct preference optimization, and continual learning consistently demonstrate the effectiveness, robustness, and efficiency of HFT. Compared with FFT, HFT not only significantly alleviates the forgetting problem, but also achieves the best performance in a series of downstream benchmarks, with an approximately 30% reduction in training time.

What problem does this paper attempt to address?

This paper mainly discusses the catastrophic forgetting problem that may occur in large language models (LLMs) after full fine-tuning (FFT), which means losing previously learned knowledge when learning new tasks. The study found that by periodically resetting some parameters, the original knowledge of LLMs can be partially restored. To this end, the paper proposes the Half Fine-Tuning (HFT) method, which updates only half of the parameters to learn new tasks, while keeping the other half unchanged to retain previous knowledge. This method does not require changing the model architecture and can be seamlessly integrated into existing fine-tuning frameworks. In experiments, HFT has shown effectiveness and robustness in scenarios such as supervised fine-tuning, direct preference optimization, and continual learning. It not only alleviates the forgetting problem but also achieves the best performance in a series of downstream tasks, while reducing training time by about 30%. Through comparative experiments, HFT significantly recovers baseline knowledge while maintaining or improving new abilities, providing a new approach to solving the catastrophic forgetting problem of LLMs. The main contributions of the paper include: 1. Discovery that resetting half of the fine-tuned parameters can partially recover the original abilities while maintaining newly learned abilities. 2. Proposal of the Half Fine-Tuning method, which randomly freezes half of the parameters during training and updates the other half, allowing the model to retain old knowledge while learning new tasks. 3. Experimental results show that HFT is effective and efficient in various training settings, making it a potential alternative to the standard fine-tuning method in the era of LLMs.

HFT: Half Fine-Tuning for Large Language Models

HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy

QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources

Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model

Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning

Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning

UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function

Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging

Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts

Less-forgetting Multi-lingual Fine-tuning

Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models

FeTT: Continual Class Incremental Learning via Feature Transformation Tuning

Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

Deconfounded Causality-aware Parameter-Efficient Fine-Tuning for Problem-Solving Improvement of LLMs

PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning

Scaling Laws for Forgetting When Fine-Tuning Large Language Models

S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

Intuitive Fine-Tuning: Towards Unifying SFT and RLHF into a Single Process

A Framework for Fine-Tuning LLMs using Heterogeneous Feedback

Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model