Abstract:The in-context learning ability of large language models (LLMs) enables them to generalize to novel downstream tasks with relatively few labeled examples. However, they require enormous computational resources to be deployed. Alternatively, smaller models can solve specific tasks if fine-tuned with enough labeled examples. These examples, however, are expensive to obtain. In pursuit of the best of both worlds, we study synthetic data generation of fine-tuning training data via fine-tuned teacher LLMs to improve the downstream performance of much smaller models. In four text classification and two text generation tasks, we find that both data generation and annotation dramatically improve the respective downstream model's performance, occasionally necessitating only a minor fraction of the original training dataset.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use large - language models (LLMs) to generate synthetic data to improve the performance of small models in downstream tasks under the condition of limited resources. Specifically, the paper explores generating or annotating training data by fine - tuning large - language models, thereby enhancing the performance of small models in text classification and natural - language - generation tasks. This method aims to combine the powerful generalization ability of large - language models and the efficient - deployment advantages of small models, especially in low - resource scenarios where only a small amount of labeled data is available. ### Main contributions of the paper: 1. **Proposed a synthetic - data - generation method**: By fine - tuning large - language models (such as GPT - NeoX with 20B parameters), generate high - quality synthetic data for training small models. 2. **Verified the effectiveness of the method**: Experiments were carried out on four text - classification tasks and two natural - language - generation tasks. The results show that using synthetic data can significantly improve the performance of small models, and sometimes only a small fraction of the original training data is required. 3. **Explored the impact of different data - generation strategies**: Studied the effects of two strategies, namely annotating existing unlabeled data and generating completely new data points, and found that the former performs better in some cases, but the latter can also effectively improve performance when there is no unlabeled data. ### Key technical details: - **Teacher Model**: Use GPT - NeoX with 20B parameters as the teacher model, and fine - tune it to be able to generate high - quality synthetic data. - **Student Model**: Use RoBERTa - Large and BART - Large as student models for text - classification and natural - language - generation tasks respectively. - **Data - generation strategies**: - **Annotate unlabeled data**: Annotate existing unlabeled data. - **Generate new data points**: Generate completely new input - output pairs. - **Experimental setup**: Experiments were carried out on multiple datasets, including SLURP, RTE, BoolQ, MultiRC, SGD, and WebNLG, covering different task types. ### Experimental results: - **Performance improvement**: In all tested tasks, the method of using synthetic - data generation significantly improved the performance of student models, especially when the amount of initial training data was small. - **Influence of data volume**: As the amount of synthetic data increases, the performance improvement gradually reaches saturation and even decreases in some tasks. - **Comparison with existing methods**: Compared with directly using large - scale models (such as GPT - 3.5 with 175B parameters), the method of generating synthetic data by fine - tuning shows better performance in some tasks. ### Conclusion: The paper shows that by fine - tuning large - language models to generate synthetic data, the performance of small models can be significantly improved under the condition of limited resources. This method is not only of great significance in low - resource scenarios but also provides a new direction for future research, especially in fields where the cost of data annotation is high.

Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs

Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations

Building a Family of Data Augmentation Models for Low-cost LLM Fine-tuning on the Cloud

Data Generation Using Large Language Models for Text Classification: An Empirical Case Study

Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis

The Potential and Limitations of Large Language Models for Text Classification through Synthetic Data Generation

A Survey on Data Synthesis and Augmentation for Large Language Models

Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification

On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey

A Practical Guide to Fine-tuning Language Models with Limited Data

Evaluating Language Models as Synthetic Data Generators

Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning

Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applications

Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM

From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data

Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models

SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning

Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs

Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

Making Large Language Models Better Data Creators