Abstract:We introduce the multiresolution recurrent neural network, which extends the sequence-to-sequence framework to model natural language generation as two parallel discrete stochastic processes: a sequence of high-level coarse tokens, and a sequence of natural language tokens. There are many ways to estimate or learn the high-level coarse tokens, but we argue that a simple extraction procedure is sufficient to capture a wealth of high-level discourse semantics. Such procedure allows training the multiresolution recurrent neural network by maximizing the exact joint log-likelihood over both sequences. In contrast to the standard log- likelihood objective w.r.t. natural language tokens (word perplexity), optimizing the joint log-likelihood biases the model towards modeling high-level abstractions. We apply the proposed model to the task of dialogue response generation in two challenging domains: the Ubuntu technical support domain, and Twitter conversations. On Ubuntu, the model outperforms competing approaches by a substantial margin, achieving state-of-the-art results according to both automatic evaluation metrics and a human evaluation study. On Twitter, the model appears to generate more relevant and on-topic responses according to automatic evaluation metrics. Finally, our experiments demonstrate that the proposed model is more adept at overcoming the sparsity of natural language and is better able to capture long-term structure.

Neural Discourse Modeling of Conversations

A Persona-Based Neural Conversation Model

Deep Reinforcement Learning for Dialogue Generation

A Neural Conversational Model

Neural Net Models for Open-Domain Discourse Coherence

Dialog context language modeling with recurrent neural networks

Evaluating Discourse Phenomena in Neural Machine Translation

A Hybrid RNN-CNN Encoder for Neural Conversation Model.

A Latent Variable Recurrent Neural Network for Discourse Relation Language Models

Modeling Coherence for Discourse Neural Machine Translation

Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading

Neural Conversation Models and How to Rein Them in: A Survey of Failures and Fixes

A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues

A Multi-Encoder Neural Conversation Model

Coherent Dialogue with Attention-based Language Models

Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring?

Multiresolution Recurrent Neural Networks: an Application to Dialogue Response Generation

Improved Document Modelling with a Neural Discourse Parser

Natural language processing models reveal neural dynamics of human conversation

Modeling Discourse Structure for Document-level Neural Machine Translation

Effective Spoken Language Labeling with Deep Recurrent Neural Networks