Abstract:Pre-trained conversation models (PCMs) have demonstrated remarkable results in task-oriented dialogue (TOD) systems. Many PCMs focus predominantly on dialogue management tasks like dialogue state tracking, dialogue generation tasks like response generation, or both. However, the existing PCMs seldom consider dialogue comprehension tasks, such as dialogue question answering and summarization tasks. These tasks allow PCMs to glean dialogue context from various angles. This observation naturally raises the question: Can the performance of downstream dialogue tasks be enhanced if a PCM is pre-trained on dialogue management, generation, and comprehension tasks? To investigate this, we proposed an Omnipotent Dialogue pre-training model (OmniDialog). It unifies these three dialogue tasks into a monolithic framework by multi-task learning, fostering inter-task communication. The pre-training corpus of OmniDialog spans $\mathbf{7}$ dialogue-focused tasks, drawing from $\mathbf{15}$ datasets and encompassing over $\mathbf{3.2}$ million dialogue utterances. To our knowledge, OmniDialog is a pioneering PCM pre-trained across dialogue management, generation, and comprehension domains. We evaluated its performance across four tasks: dialogue summarization, end-to-end dialogue modeling, dialogue state tracking, and intent classification. The results underscore its efficacy in domain transfer learning, low-resource, and full-dataset scenarios. Furthermore, to glean a nuanced understanding of OmniDialog's strengths and potential pitfalls, we designed a fine-grained analysis framework for dialogue-centric tasks. Experimental results show that the OmniDialog is good at hard samples, such as long dialogues and lengthy responses.

Dialogue Sentiment Analysis Based on Dialogue Structure Pre-training

Semantic-based Pre-training for Dialogue Understanding

DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization

Chinese Dialogue Analysis Using Multi-Task Learning Framework

DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder

Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization

Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning

SPECTRUM: Speaker-Enhanced Pre-Training for Long Dialogue Summarization

OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System

Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning.

DeepStruct: Pretraining of Language Models for Structure Prediction.

Sentiment-aware Multimodal Pre-Training for Multimodal Sentiment Analysis

Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure

FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue

Advances in Multi-turn Dialogue Comprehension: A Survey

Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks

STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension

Post-Training Dialogue Summarization using Pseudo-Paraphrasing

Integrating BERT Embeddings and BiLSTM for Emotion Analysis of Dialogue

SentiLR: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis

Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis