LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues

Joe Stacey,Jianpeng Cheng,John Torr,Tristan Guigue,Joris Driesen,Alexandru Coca,Mark Gaynor,Anders Johannsen

2024-05-04

Abstract:Spurred by recent advances in Large Language Models (LLMs), virtual assistants are poised to take a leap forward in terms of their dialogue capabilities. Yet a major bottleneck to achieving genuinely transformative task-oriented dialogue capabilities remains the scarcity of high quality data. Existing datasets, while impressive in scale, have limited domain coverage and contain few genuinely challenging conversational phenomena; those which are present are typically unlabelled, making it difficult to assess the strengths and weaknesses of models without time-consuming and costly human evaluation. Moreover, creating high quality dialogue data has until now required considerable human input, limiting both the scale of these datasets and the ability to rapidly bootstrap data for a new target domain. We aim to overcome these issues with LUCID, a modularised and highly automated LLM-driven data generation system that produces realistic, diverse and challenging dialogues. We use LUCID to generate a seed dataset of 4,277 conversations across 100 intents to demonstrate its capabilities, with a human review finding consistently high quality labels in the generated data.

Computation and Language

What problem does this paper attempt to address?

The paper mainly addresses the following issues:

1. **Scarcity of high-quality dialogue data**: Although existing dialogue datasets are large in scale, they are limited in domain coverage and contain few complex dialogue phenomena, which are often unmarked, making it difficult to evaluate the model's capabilities.

2. **High cost of dialogue data generation**: Creating high-quality dialogue data often requires a significant amount of manual effort, which limits the scale of the datasets and their ability to quickly adapt to new domains.

To solve the above problems, the research team proposed LUCID (LLM-Generated Utterances for Complex and Interesting Dialogues), a modular, highly automated data generation system based on large language models, designed to produce realistic, diverse, and challenging dialogue data. LUCID decomposes the data generation process through a series of modular large language model calls, ensuring that each step can be accurately executed by the model, thereby achieving large-scale, high-quality data generation. Additionally, LUCID employs a series of validation mechanisms to ensure the quality of the generated data, including multiple rounds of validation to reduce errors and inconsistencies. In this way, LUCID not only generates more diverse and challenging dialogue data but also significantly reduces the cost and time consumption of the generation process.