Abstract:The rise of large language models (LLMs) has created a significant disparity: industrial research labs with their computational resources, expert teams, and advanced infrastructures, can effectively fine-tune LLMs, while individual developers and small organizations face barriers due to limited resources. In this paper, we aim to bridge this gap by presenting a comprehensive study on supervised fine-tuning of LLMs using instruction-tuning datasets spanning diverse knowledge domains and skills. We focus on small-sized LLMs (3B to 7B parameters) for their cost-efficiency and accessibility. We explore various training configurations and strategies across four open-source pre-trained models. We provide detailed documentation of these configurations, revealing findings that challenge several common training practices, including hyperparameter recommendations from TULU and phased training recommended by Orca. Key insights from our work include: (i) larger batch sizes paired with lower learning rates lead to improved model performance on benchmarks such as MMLU, MTBench, and Open LLM Leaderboard; (ii) early-stage training dynamics, such as lower gradient norms and higher loss values, are strong indicators of better final model performance, enabling early termination of sub-optimal runs and significant computational savings; (iii) through a thorough exploration of hyperparameters like warmup steps and learning rate schedules, we provide guidance for practitioners and find that certain simplifications do not compromise performance; and (iv) we observed no significant difference in performance between phased and stacked training strategies, but stacked training is simpler and more sample efficient. With these findings holding robustly across datasets and models, we hope this study serves as a guide for practitioners fine-tuning small LLMs and promotes a more inclusive environment for LLM research.

Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning

Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

Deconfounded Causality-aware Parameter-Efficient Fine-Tuning for Problem-Solving Improvement of LLMs

Parameter-efficient fine-tuning of large-scale pre-trained language models

Towards a Unified View of Parameter-Efficient Transfer Learning

Full Parameter Fine-tuning for Large Language Models with Limited Resources

Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs

Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

Aligner: One Global Token is Worth Millions of Parameters when Aligning Large Language Models

QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources

Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

ADT: an Additive Delta-Tuning Approach for Parameter-Efficient Tuning in Pre-Trained Language Models

Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model

Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models

Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning