Abstract:We introduce the multiresolution recurrent neural network, which extends the sequence-to-sequence framework to model natural language generation as two parallel discrete stochastic processes: a sequence of high-level coarse tokens, and a sequence of natural language tokens. There are many ways to estimate or learn the high-level coarse tokens, but we argue that a simple extraction procedure is sufficient to capture a wealth of high-level discourse semantics. Such procedure allows training the multiresolution recurrent neural network by maximizing the exact joint log-likelihood over both sequences. In contrast to the standard log- likelihood objective w.r.t. natural language tokens (word perplexity), optimizing the joint log-likelihood biases the model towards modeling high-level abstractions. We apply the proposed model to the task of dialogue response generation in two challenging domains: the Ubuntu technical support domain, and Twitter conversations. On Ubuntu, the model outperforms competing approaches by a substantial margin, achieving state-of-the-art results according to both automatic evaluation metrics and a human evaluation study. On Twitter, the model appears to generate more relevant and on-topic responses according to automatic evaluation metrics. Finally, our experiments demonstrate that the proposed model is more adept at overcoming the sparsity of natural language and is better able to capture long-term structure.

Dialog context language modeling with recurrent neural networks

Sequential Dialogue Context Modeling for Spoken Language Understanding

Neural Discourse Modeling of Conversations

A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks

Contextual Recurrent Units for Cloze-style Reading Comprehension

A Latent Variable Recurrent Neural Network for Discourse Relation Language Models

Multi-level Gated Recurrent Neural Network for Dialog Act Classification

Recurrent Neural Network Language Model with Part-of-speech for Mandarin Speech Recognition.

How To Make Context More Useful? An Empirical Study On Context-Aware Neural Conversational Models

Global context-dependent recurrent neural network language model with sparse feature learning

How recurrent networks implement contextual processing in sentiment analysis

Long-Short Range Context Neural Networks for Language Modeling

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Chinese Semantic Role Labeling with Bidirectional Recurrent Neural Networks

Multiresolution Recurrent Neural Networks: an Application to Dialogue Response Generation

Dialog State Tracking Using Long Short-Term Memory Neural Networks.

Recurrent Memory Networks for Language Modeling

Discriminative method for recurrent neural network language models

Detecting Interrogative Utterances with Recurrent Neural Networks

Contextual LSTM (CLSTM) models for Large scale NLP tasks

Improving Accented Mandarin Speech Recognition by Using Recurrent Neural Network Based Language Model Adaptation