Abstract:To induce desired behaviors in large language models (LLMs) for interaction-driven tasks, the instruction-tuning stage typically trains LLMs on instruction-response pairs using the next-token prediction (NTP) loss. Previous work aiming to improve instruction-tuning performance often emphasizes the need for higher-quality supervised fine-tuning (SFT) datasets, which typically involves expensive data filtering with proprietary LLMs or labor-intensive data generation by human annotators. However, these approaches do not fully leverage the datasets' intrinsic properties, resulting in high computational and labor costs, thereby limiting scalability and performance gains. In this paper, we propose SFTMix, a novel recipe that elevates instruction-tuning performance beyond the conventional NTP paradigm, without the need for well-curated datasets. Observing that LLMs exhibit uneven confidence across the semantic representation space, we argue that examples with different confidence levels should play distinct roles during the instruction-tuning process. Based on this insight, SFTMix leverages training dynamics to identify examples with varying confidence levels, then applies a Mixup-based regularization to mitigate overfitting on confident examples while propagating supervision signals to improve learning on relatively unconfident ones. This approach enables SFTMix to significantly outperform NTP across a wide range of instruction-following and healthcare domain-specific SFT tasks, demonstrating its adaptability to diverse LLM families and scalability to datasets of any size. Comprehensive ablation studies further verify the robustness of SFTMix's design choices, underscoring its versatility in consistently enhancing performance across different LLMs and datasets in broader natural language processing applications.

Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction

Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging

SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe

Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment

XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts

Maybe Only 0.5 Training Data Instruction Tuning

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition

Demystifying Instruction Mixing for Fine-tuning Large Language Models

Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models

CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model

LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms

UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function

Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation

Enhancing Task Performance in Continual Instruction Fine-tuning Through Format Uniformity

Instruction Mining: Instruction Data Selection for Tuning Large Language Models

CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions

Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging

From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

Federated Data-Efficient Instruction Tuning for Large Language Models

Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models

Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model