Abstract:Large language models (LLMs) with enormous pre-training tokens and parameters emerge diverse abilities, including math reasoning, code generation, and instruction following. These abilities are further enhanced by supervised fine-tuning (SFT). While the open-source community has explored ad-hoc SFT for enhancing individual capabilities, proprietary LLMs exhibit versatility across various skills. Therefore, understanding the facilitation of multiple abilities via SFT is paramount. In this study, we specifically focuses on the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during SFT. We propose four intriguing research questions to explore the association between model performance and various factors including data amount, composition ratio, model size and SFT strategies. Our experiments reveal that distinct capabilities scale differently and larger models generally show superior performance with same amount of data. Mathematical reasoning and code generation consistently improve with increasing data amount, whereas general abilities plateau after roughly a thousand samples. Moreover, we observe data composition appears to enhance various abilities under limited data conditions, yet can lead to performance conflicts when data is plentiful. Our findings also suggest the amount of composition data influences performance more than the composition ratio. In analysis of SFT strategies, we find that sequentially learning multiple skills risks catastrophic forgetting. Our proposed Dual-stage Mixed Fine-tuning (DMT) strategy offers a promising solution to learn multiple abilities with different scaling patterns.

What problem does this paper attempt to address?

The paper primarily investigates the performance capabilities of Large Language Models (LLMs) during the Supervised Fine-Tuning (SFT) phase, with a particular focus on how the composition of data affects abilities such as mathematical reasoning, code generation, and general alignment with human intent. The researchers conducted an in-depth analysis through four specific research questions: 1. **The relationship between different SFT abilities and the amount of data**: The study examines how mathematical reasoning, coding, and general abilities perform as the amount of SFT data changes, finding that different abilities have different scaling patterns, and larger models generally perform better with the same amount of data. 2. **Capability conflicts during multi-task learning**: Whether performance conflicts arise when combining these three abilities for SFT. The results indicate that under low-resource conditions, multi-task learning of various abilities can improve, while under high-resource conditions, it may decline, especially in smaller models. 3. **Key factors causing performance conflicts**: The study explores the impact of data quantity and data ratio on capability conflicts, finding that direct data quantity has a greater impact on each ability than data ratio. In particular, when data comes from different but similar tasks, changes in data ratio can lead to significant performance fluctuations. 4. **The impact of different SFT strategies**: Experiments compared strategies such as multi-task learning, sequential training, mixed sequential training, and the proposed Dual-Stage Mixed Tuning (DMT), finding that the DMT strategy can effectively mitigate performance conflicts in multi-task learning and catastrophic forgetting in sequential training, achieving a balance between general and specialized abilities. The research utilized three benchmark datasets: GSM8K for mathematical reasoning, HumanEval for code generation, and MT-Bench for general human alignment ability assessment. Additionally, the study considered models of different scales, from LLaMA 7B to 33B, and explored four different SFT strategies, including multi-task learning, sequential training, mixed sequential training, and the proposed Dual-Stage Mixed Tuning (DMT). The experimental results revealed the scaling patterns of different abilities with increasing data volume and the performance differences across various model scales, while also demonstrating the advantages of the DMT strategy in handling multi-task learning and avoiding catastrophic forgetting.

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition

Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities

An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning

VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs

Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model

Empirical Insights on Fine-Tuning Large Language Models for Question-Answering

Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability

Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction

Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning

Optimizing Language Model's Reasoning Abilities with Weak Supervision

ReFT: Reasoning with Reinforced Fine-Tuning

Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models

Dynamics of Instruction Tuning: Each Ability of Large Language Models Has Its Own Growth Pace

When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging

Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning

Deconfounded Causality-aware Parameter-Efficient Fine-Tuning for Problem-Solving Improvement of LLMs

Improving Large Language Model Fine-tuning for Solving Math Problems