Abstract:Pre-training on large-scale open-domain dialogue data can substantially improve the performance of dialogue models. However, the pre-trained dialogue model's ability to utilize long-range context is limited due to the scarcity of long-turn dialogue sessions. Most dialogues in existing pre-training corpora contain fewer than three turns of dialogue. To alleviate this issue, we propose the Retrieve, Reorganize and Rescale framework (Re$^3$Dial), which can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. Given a short-turn session, Re$^3$Dial first employs a session retriever to retrieve coherent consecutive sessions. To this end, we train the retriever to capture semantic and discourse relations within multi-turn dialogues through contrastive training. Next, Re$^3$Dial samples a session from retrieved results following a diversity sampling strategy, which is designed to penalize repetitive or generic sessions. A longer session is then derived by concatenating the original session and the sampled session. By repeating the above process, Re$^3$Dial can yield a coherent long-turn dialogue. Extensive experiments on multiple multi-turn dialogue benchmarks demonstrate that Re$^3$Dial significantly improves the dialogue model's ability to utilize long-range context and thus generate more sensible and informative responses. Finally, we build a toolkit for efficiently rescaling conversations with Re$^3$Dial, which enables us to construct a corpus containing 1B Chinese dialogue sessions with 11.3 turns on average (5$\times$ longer than the original corpus). Our retriever model, code, and data is publicly available at \url{<a class="link-external link-https" href="https://github.com/thu-coai/Re3Dial" rel="external noopener nofollow">this https URL</a>}.

Building Context-Related Dialogue Systems Based on Chinese-Script-Dialogue Corpus

An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation

OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts

A Manually Annotated Chinese Corpus for Non-task-oriented Dialogue Systems

The JDDC Corpus: A Large-Scale Multi-Turn Chinese Dialogue Dataset forE-commerce Customer Service

Context Directed Speech Recognition in Dialogue Systems

The JDDC Corpus: A Large-Scale Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Service

Automatic Construction of Discourse Corpora for Dialogue Translation

XDailyDialog: A Multilingual Parallel Dialogue Corpus

CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generation

Re$^3$Dial: Retrieve, Reorganize and Rescale Dialogue Corpus for Long-Turn Open-Domain Dialogue Pre-training

A Pilot Study on Dialogue-Level Dependency Parsing for Chinese

An Empirical Study on Context Length for Open-Domain Dialog Generation

Multi-turn dialogue comprehension from a topic-aware perspective

Searching for Snippets of Open-Domain Dialogue in Task-Oriented Dialogue Datasets

Dialogue Generation Model with Hierarchical Encoding and Semantic Segmentation of Dialogue Context

CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

JDDC 2.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization

When to Talk: Chatbot Controls the Timing of Talking during Multi-turn Open-domain Dialogue Generation

Deep context modeling for multi-turn response selection in dialogue systems