Abstract:Multi-turn response selection is a major task in building intelligent dialogue systems. Most existing works focus on modeling the semantic relationship between the utterances and the candidate response with neural networks like RNNs and various attention mechanisms. In this paper, we study how to leverage the advantage of pre-trained language models (PTMs) to multi-turn response selection in retrieval-based chatbots. We propose a deep context modeling architecture (DCM) for multi-turn response selection by utilizing BERT as the context encoder. DCM is formulated as a four-module architecture, namely contextual encoder, utterance-to-response interaction, features aggregation, and response selection. Moreover, in DCM, we introduce the next utterance prediction as a pre-training scheme based on BERT, aiming to adapt general BERT to accommodate the inherent context continuity underlying the multi-turn dialogue. Taking BERT as the backbone encoder, we then investigate a variety of strategies to perform response selection with comprehensive comparisons. Empirical results on three public datasets from two different languages show that our proposed model outperforms existing promising models significantly, pushing recall to 86.8% (+5.2% improvement over BERT) on Ubuntu Dialogue corpus, recall to 68.5% (+6.4% improvement over BERT) on E-Commerce Dialogue corpus, MAP and MRR to 61.6% and 64.9% respectively (+2.3% and 1.8% improvement over BERT) on Douban Conversation corpus, achieving new state-of-the-art performance for multi-turn response selection.

How To Make Context More Useful? An Empirical Study On Context-Aware Neural Conversational Models

How to Represent Context Better? an Empirical Study on Context Modeling for Multi-turn Response Selection.

Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation

Coupled Context Modeling For Deep Chit-Chat: Towards Conversations Between Human And Computer

Neural Contextual Conversation Learning with Labeled Question-Answering Pairs

Context-aware Natural Language Generation for Spoken Dialogue Systems.

When Less is More: Using Less Context Information to Generate Better Utterances in Group Conversations.

Context-Controlled Topic-Aware Neural Response Generation for Open-Domain Dialog Systems

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

An Empirical Study on Context Length for Open-Domain Dialog Generation

A Survey on Response Selection for Retrieval-based Dialogues.

Context Retrieval via Normalized Contextual Latent Interaction for Conversational Agent

CA-BERT: Leveraging Context Awareness for Enhanced Multi-Turn Chat Interaction

Unsupervised Context Rewriting for Open Domain Conversation

Deep context modeling for multi-turn response selection in dialogue systems

Towards Making the Most of Context in Neural Machine Translation

Diving Deep into Context-Aware Neural Machine Translation

Evaluating and Improving Context Attention Distribution on Multi-Turn Response Generation using Self-Contained Distractions

ContextMate: a Context-Aware Smart Agent for Efficient Data Analysis

Cluster Analysis Based on Contextual Features Extraction for Conversational Corpus

Exploring the importance of context and embeddings in neural NER models for task-oriented dialogue systems