Abstract:During the pretraining phase, large language models (LLMs) acquire vast amounts of knowledge from extensive text corpora. Nevertheless, in later stages such as fine-tuning and inference, the model may encounter knowledge not covered in the initial training, which can lead to hallucinations and degraded performance. This issue has a profound impact on the model's capabilities, as it will inevitably face out-of-scope knowledge after pretraining. Furthermore, fine-tuning is often required to adapt LLMs to domain-specific tasks. However, this phenomenon limits the model's ability to learn and integrate new information during fine-tuning. The effectiveness of fine-tuning largely depends on the type of knowledge involved. Existing research suggests that fine-tuning the model on partially mastered knowledge-for instance, question-answer pairs where the model has a chance of providing correct responses under non-greedy decoding-can enable the model to acquire new knowledge while mitigating hallucination. Notably, this approach can still lead to the forgetting of fully mastered knowledge, constraining the fine-tuning dataset to a narrower range and limiting the model's overall potential for improvement. Given the model's intrinsic reasoning abilities and the interconnectedness of different knowledge areas, it is likely that as the model's capacity to utilize existing knowledge improves during fine-tuning, previously unmastered knowledge may become more understandable. To explore this hypothesis, we conducted experiments and, based on the results, proposed a two-stage fine-tuning strategy. This approach not only improves the model's overall test accuracy and knowledge retention but also preserves its accuracy on previously mastered content. When fine-tuning on the WikiQA dataset, our method increases the amount of knowledge acquired by the model in this stage by 24%.

Refine Large Language Model Fine-tuning via Instruction Vector

Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models

Dissecting Learning and Forgetting in Language Model Finetuning

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Revisiting Catastrophic Forgetting in Large Language Model Tuning

Scaling Laws for Forgetting When Fine-Tuning Large Language Models

Learning or Self-aligning? Rethinking Instruction Fine-tuning

Can LLMs Learn New Concepts Incrementally without Forgetting?

Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Models

Fine-tuning can cripple your foundation model; preserving features may be the solution

Exploring Forgetting in Large Language Model Pre-Training

Less-forgetting Multi-lingual Fine-tuning

Demystifying Language Model Forgetting with Low-rank Example Associations

HFT: Half Fine-Tuning for Large Language Models

Separable Mixture of Low-Rank Adaptation for Continual Visual Instruction Tuning

Why Fine-Tuning Struggles with Forgetting in Machine Unlearning? Theoretical Insights and a Remedial Approach

Chained Tuning Leads to Biased Forgetting

Large Language Models with Controllable Working Memory

Dissecting Fine-Tuning Unlearning in Large Language Models

Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning

From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data