Abstract:Large language models (LLMs) have enabled a range of applications in zero-shot and few-shot learning settings, including the generation of synthetic datasets for training and testing. However, to reliably use these synthetic datasets, it is essential to understand how representative they are of real-world data. We investigate this by assessing the effectiveness of generating synthetic data through LLM and using it as a benchmark for various NLP tasks. Our experiments across six datasets, and three different tasks, show that while synthetic data can effectively capture performance of various methods for simpler tasks, such as intent classification, it falls short for more complex tasks like named entity recognition. Additionally, we propose a new metric called the bias factor, which evaluates the biases introduced when the same LLM is used to both generate benchmarking data and to perform the tasks. We find that smaller LLMs exhibit biases towards their own generated data, whereas larger models do not. Overall, our findings suggest that the effectiveness of synthetic data as a benchmark varies depending on the task, and that practitioners should rely on data generated from multiple larger models whenever possible.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the effectiveness of synthetic data generated by large - language models (LLMs) as a benchmark, especially in predicting the performance of LLMs on various natural - language - processing (NLP) tasks in a zero - sample - learning setting. Specifically, the researchers focus on the following aspects: 1. **Effectiveness of synthetic data**: The researchers evaluate the effectiveness of synthetic data as a benchmark through six datasets and three different tasks (intent detection, text similarity, named - entity recognition). They mainly evaluate from two perspectives: - **Absolute performance**: Measure the performance difference of the model on the real dataset and the synthetic dataset. - **Relative performance**: Evaluate the relative ranking change of different methods on the real dataset and the synthetic dataset. 2. **Quantification of potential bias**: The researchers propose a new metric - the bias factor - to quantify the bias introduced when the same LLM is used to generate synthetic data and perform tasks. The bias factor takes into account the difference between the performance of the LLM on its own - generated data and its performance on data generated by other LLMs. 3. **Applicability of different tasks**: The study finds that for simpler tasks (such as intent detection), synthetic data can effectively capture the performance of the model; while for more complex tasks (such as named - entity recognition), the representativeness of synthetic data is poorer. 4. **Performance of models of different scales**: The study also finds that smaller LLMs show higher bias when using their own - generated data, while larger LLMs have no obvious bias phenomenon. In summary, the main contribution of this paper is to systematically evaluate the effectiveness of synthetic data generated by LLM as a benchmark and propose a new metric to quantify potential bias. The research results show that synthetic data performs better in simple tasks but poorer in complex tasks, and LLMs of different scales also perform differently when generating synthetic data. Therefore, the study recommends that in practical applications, synthetic data generated by multiple larger - scale LLMs should be used as much as possible to obtain more robust and representative benchmarks.

Efficacy of Synthetic Data as a Benchmark

Evaluating Language Models as Synthetic Data Generators

Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs

Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs

A synthetic data approach for domain generalization of NLI models

Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

Better Synthetic Data by Retrieving and Transforming Existing Datasets

Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images

On the Diversity of Synthetic Data and its Impact on Training Large Language Models

A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models

Best Practices and Lessons Learned on Synthetic Data

On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey

Under the Surface: Tracking the Artifactuality of LLM-Generated Data

Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models

HLB: Benchmarking LLMs' Humanlikeness in Language Use

Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

Data Generation Using Large Language Models for Text Classification: An Empirical Case Study

Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction