Abstract:In recent times, sequence-to-sequence (seq2seq) models have gained a lot of popularity and provide state-of-the-art performance in a wide variety of tasks such as machine translation, headline generation, text summarization, speech to text conversion, and image caption generation. The underlying framework for all these models is usually a deep neural network comprising an encoder and a decoder. Although simple encoder-decoder models produce competitive results, many researchers have proposed additional improvements over these sequence-to-sequence models, e.g., using an attention-based model over the input, pointer-generation models, and self-attention models. However, such seq2seq models suffer from two common problems: 1) exposure bias and 2) inconsistency between train/test measurement. Recently, a completely novel point of view has emerged in addressing these two problems in seq2seq models, leveraging methods from reinforcement learning (RL). In this survey, we consider seq2seq problems from the RL point of view and provide a formulation combining the power of RL methods in decision-making with sequence-to-sequence models that enable remembering long-term memories. We present some of the most recent frameworks that combine concepts from RL and deep neural networks and explain how these two areas could benefit from each other in solving complex seq2seq tasks. Our work aims to provide insights into some of the problems that inherently arise with current approaches and how we can address them with better RL models. We also provide the source code for implementing most of the RL models discussed in this paper to support the complex task of abstractive text summarization.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on two common issues in the sequence - to - sequence (seq2seq) model: 1. **Exposure Bias**: During the training process, the decoder depends on the real previous - step output (i.e., the ground truth) to generate the next - step action. However, during the testing stage, since there is no real previous - step output, the decoder can only rely on the previous - step output generated by the model itself. This inconsistency between training and testing leads to exposure bias and may cause error accumulation. 2. **Inconsistency Between Train/Test Measurement**: The seq2seq model is usually trained using cross - entropy loss, but during the testing stage, the performance evaluation of the model usually uses discrete and non - differentiable metrics (such as BLEU, ROUGE, etc.). This mismatch between the training objective and the testing metric may lead to poor performance of the model in practical applications. To solve these problems, the paper proposes a new perspective, that is, using the reinforcement learning (RL) method to improve the seq2seq model. Specifically, the paper explores how to combine RL methods to enable the seq2seq model to better remember long - term memories and be more flexible in the decision - making process. In this way, exposure bias can be reduced, and the model can directly optimize the evaluation metrics used in the testing stage during training. The main contributions of the paper include: - Providing a comprehensive review of RL methods used for deep learning, especially for seq2seq model training. - Summarizing the challenges, advantages, and disadvantages of using different RL methods for seq2seq training. - Providing guidance on improving specific RL methods to obtain better and smoother seq2seq model training. - Open - sourcing a library for implementing complex seq2seq models, supporting the use of different RL techniques, and providing experimental results showing the degree of improvement of RL algorithms on current seq2seq models. Through these contributions, the paper aims to expand the capabilities of the seq2seq model and explore the challenges faced in applying these methods in the context of deep learning.

Deep Reinforcement Learning For Sequence to Sequence Models

Deep Reinforcement Learning for NLP.

Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback

ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation

Sequence-to-Sequence ASR Optimization via Reinforcement Learning

Guided Reinforcement Learning via Sequence Learning

Neural Sequence-to-Sequence Modeling with Attention by Leveraging Deep Learning Architectures for Enhanced Contextual Understanding in Abstractive Text Summarization

Robust Reinforcement Learning Objectives for Sequential Recommender Systems

Teach and Explore: A Multiplex Information-guided Effective and Efficient Reinforcement Learning for Sequential Recommendation

TaskSum: Task-Driven Extractive Text Summarization for Long News Documents Based on Reinforcement Learning.

Generating Text with Deep Reinforcement Learning

A Short Survey On Memory Based Reinforcement Learning

Exploring Reinforcement Learning Methods for Multiple Sequence Alignment: A Brief Review

Deep Reinforcement Learning: An Overview

Automatic Curriculum Learning For Deep RL: A Short Survey

Optimizing Attention for Sequence Modeling Via Reinforcement Learning.

Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation

Structure in Deep Reinforcement Learning: A Survey and Open Problems

Automated Reinforcement Learning: An Overview

Deep Reinforcement Learning: A Brief Survey

Video Summarisation by Classification with Deep Reinforcement Learning