Abstract:Pre-trained conversation models (PCMs) have demonstrated remarkable results in task-oriented dialogue (TOD) systems. Many PCMs focus predominantly on dialogue management tasks like dialogue state tracking, dialogue generation tasks like response generation, or both. However, the existing PCMs seldom consider dialogue comprehension tasks, such as dialogue question answering and summarization tasks. These tasks allow PCMs to glean dialogue context from various angles. This observation naturally raises the question: Can the performance of downstream dialogue tasks be enhanced if a PCM is pre-trained on dialogue management, generation, and comprehension tasks? To investigate this, we proposed an Omnipotent Dialogue pre-training model (OmniDialog). It unifies these three dialogue tasks into a monolithic framework by multi-task learning, fostering inter-task communication. The pre-training corpus of OmniDialog spans $\mathbf{7}$ dialogue-focused tasks, drawing from $\mathbf{15}$ datasets and encompassing over $\mathbf{3.2}$ million dialogue utterances. To our knowledge, OmniDialog is a pioneering PCM pre-trained across dialogue management, generation, and comprehension domains. We evaluated its performance across four tasks: dialogue summarization, end-to-end dialogue modeling, dialogue state tracking, and intent classification. The results underscore its efficacy in domain transfer learning, low-resource, and full-dataset scenarios. Furthermore, to glean a nuanced understanding of OmniDialog's strengths and potential pitfalls, we designed a fine-grained analysis framework for dialogue-centric tasks. Experimental results show that the OmniDialog is good at hard samples, such as long dialogues and lengthy responses.

Pretraining Methods for Dialog Context Representation Learning

DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder

Dialogue-oriented Pre-training

Structural Pre-training for Dialogue Comprehension

Multimodal Pretraining from Monolingual to Multilingual

When Does Further Pre-training MLM Help? an Empirical Study on Task-Oriented Dialog Pre-training

Training Neural Response Selection for Task-Oriented Dialogue Systems

Improving Knowledge Graph Representation Learning by Structure Contextual Pre-training

Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization

Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks

Fine-grained Post-training for Improving Retrieval-based Dialogue Systems

DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation

Multi-Granularity Representations of Dialog

OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System

Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation

Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues

Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System

Semantic-based Pre-training for Dialogue Understanding

Deep context modeling for multi-turn response selection in dialogue systems

DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization

Investigating the Impact of Pre-trained Language Models on Dialog Evaluation