Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging

Yiming Ju,Ziyi Ni,Xingrun Xing,Zhixiong Zeng,hanyu Zhao,Siqi Fan,Zheng Zhang
2024-10-01
Abstract:Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. In this work, we demonstrate that the order of training data can lead to significant training imbalances, potentially resulting in performance degradation. Consequently, we propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders, thereby enhancing the overall effectiveness of SFT. Additionally, we introduce a novel technique, "parameter-selection merging," which outperforms traditional weighted-average methods on five datasets. Further, through analysis and ablation studies, we validate the effectiveness of our method and identify the sources of performance improvements.
Computation and Language,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The paper attempts to address the significant training imbalance caused by the order of training data during the supervised fine-tuning (SFT) process of large language models (LLMs). This imbalance can lead to a decline in model performance. The paper proposes mitigating this imbalance by merging SFT models fine-tuned with different data orders and introduces a new parameter merging technique called "parameter selective merging." This method outperforms the traditional weighted average method across 5 datasets. Through analysis and ablation studies, the effectiveness of this method is validated, and the specific sources of performance improvement are identified. Additionally, experimental results show that the merged model performs better on various tasks compared to a single SFT model. In particular, in multi-task merging scenarios, the parameter selective merging method significantly outperforms the weighted average method.