Abstract:In recent years, with the rapid advancements in Natural Language Processing (NLP) technologies, large models have become widespread. Traditional reinforcement learning algorithms have also started experimenting with language models to optimize training. However, they still fundamentally rely on the Markov Decision Process (MDP) for reinforcement learning, and do not fully exploit the advantages of language models for dealing with long sequences of problems. The Decision Transformer (DT) introduced in 2021 is the initial effort to completely transform the reinforcement learning problem into a challenge within the NLP domain. It attempts to use text generation techniques to create reinforcement learning trajectories, addressing the issue of finding optimal trajectories. However, the article places the training trajectory data of reinforcement learning directly into a basic language model for training. Its aim is to predict the entire trajectory, encompassing state and reward information. This approach deviates from the reinforcement learning training objective of finding the optimal action. Furthermore, it generates redundant information in the output, impacting the final training effectiveness of the agent. This paper proposes a more reasonable network model structure, the Action-Translator Transformer (ATT), to predict only the next action of the agent. This makes the language model more interpretable for the reinforcement learning problem. We test our model in simulated gaming scenarios and compare it with current mainstream methods in the offline reinforcement learning field. Based on the presented experimental results, our model demonstrates superior performance. We hope that introducing this model will inspire new ideas and solutions for combining language models and reinforcement learning, providing fresh perspectives for offline reinforcement learning research.

Non-autoregressive Transformer by Position Learning

PNAT: Non-autoregressive Transformer by Position Learning

On the Learning of Non-Autoregressive Transformers.

Hybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation

Non-Autoregressive Translation by Learning Target Categorical Codes.

Recurrent Positional Embedding for Neural Machine Translation.

Unlocking the Power of GANs in Non-Autoregressive Text Generation

Directed Acyclic Transformer for Non-Autoregressive Machine Translation.

Cascaded Text Generation with Markov Transformers

An Augmented Transformer Architecture for Natural Language Generation Tasks

Non-iterative Parallel Text Generation via Glancing Transformer

Glancing Transformer for Non-Autoregressive Neural Machine Translation.

Optimizing Non-Autoregressive Transformers with Contrastive Learning

Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation

Semi-Autoregressive Neural Machine Translation

Non-Autoregressive Text Generation with Pre-trained Language Models

Non-autoregressive Translation with Dependency-Aware Decoder

Generalizable autoregressive modeling of time series through functional narratives

Non-Autoregressive Machine Translation with Auxiliary Regularization

RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer

Optimizing Reinforcement Learning Using a Generative Action-Translator Transformer