Abstract:Large language models (LLMs) have become a dominant and important tool for NLP researchers in a wide range of tasks. Today, many researchers use LLMs in synthetic data generation, task evaluation, fine-tuning, distillation, and other model-in-the-loop research workflows. However, challenges arise when using these models that stem from their scale, their closed source nature, and the lack of standardized tooling for these new and emerging workflows. The rapid rise to prominence of these models and these unique challenges has had immediate adverse impacts on open science and on the reproducibility of work that uses them. In this paper, we introduce DataDreamer, an open source Python library that allows researchers to write simple code to implement powerful LLM workflows. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science and reproducibility. The library and documentation are available at <a class="link-external link-https" href="https://github.com/datadreamer-dev/DataDreamer" rel="external noopener nofollow">this https URL</a> .

What problem does this paper attempt to address?

The paper aims to address the challenges faced when applying large language models (LLMs) in natural language processing (NLP) research, particularly concerning reproducibility and open science issues related to synthetic data generation, task evaluation, fine-tuning, and alignment processes. Specifically: 1. **Simplifying Complex Workflows**: Many popular LLMs like GPT-4 are closed-source and require access via remote APIs, making local execution technically complex and costly. Additionally, existing prompt paradigms perform inconsistently across different models, configurations, and environments, making it difficult for researchers to share, reproduce, extend, and compare research results. 2. **Supporting Multiple Workflows**: The paper introduces DataDreamer, an open-source Python library designed to implement advanced LLM workflows, including synthetic data generation, fine-tuning, instruction tuning, and model alignment tasks. It simplifies the implementation of these tasks through standardized interfaces, reducing the technical complexity of switching between different models. 3. **Improving Reproducibility**: DataDreamer aims to promote open science and the reproducibility of research results by providing features such as automatic caching, interruptibility, and recovery, making the implementation of multi-stage workflows simpler and capable of automatically generating fingerprint information that helps verify experimental consistency. 4. **Integrating Optimization Techniques**: The tool also integrates techniques such as quantization, adapter optimization, and multi-GPU training, thereby improving the efficiency of large-scale data and model processing. In summary, DataDreamer is dedicated to simplifying the implementation process of LLM-related research tasks and ensuring that these studies can be easily reproduced and extended, thereby advancing the field of NLP.

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

Under the Surface: Tracking the Artifactuality of LLM-Generated Data

Data-Prep-Kit: getting your data ready for LLM application development

On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey

LLMs for science: Usage for code generation and data analysis

LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

ToolBridge: An Open-Source Dataset to Equip LLMs with External Tool Capabilities

MLLM-DataEngine: An Iterative Refinement Approach for MLLM

DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models

Making Large Language Models Better Data Creators

Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science

The Evolution of LLM Adoption in Industry Data Curation Practices

DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles

Large Language Models for Data Annotation and Synthesis: A Survey

Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes

SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks

LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives

Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation

Exploring LLMs as a Source of Targeted Synthetic Textual Data to Minimize High Confidence Misclassifications

LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents

LLM Attributor: Interactive Visual Attribution for LLM Generation