Abstract:Model-based reinforcement learning (RL) is regarded as a promising approach to tackle the challenges that hinder model-free RL. The success of model-based RL hinges critically on the quality of the predicted dynamic models. However, for many real-world tasks involving high-dimensional state spaces, current dynamics prediction models show poor performance in long-term prediction. To that end, we propose a novel two-branch neural network architecture with multi-timescale memory augmentation to handle long-term and short-term memory differently. Specifically, we follow previous works to introduce a recurrent neural network architecture to encode history observation sequences into latent space, characterizing the long-term memory of agents. Different from previous works, we view the most recent observations as the short-term memory of agents and employ them to directly reconstruct the next frame to avoid compounding error. This is achieved by introducing a self-supervised optical flow prediction structure to model the action-conditional feature transformation at pixel level. The reconstructed observation is finally augmented by the long-term memory to ensure semantic consistency. Experimental results show that our approach is able to generate visually-realistic long-term predictions in DeepMind maze navigation games, and outperforms the prevalent state-of-the-art methods in prediction accuracy by a large margin. Furthermore, we also evaluate the usefulness of our world model by using the predicted frames to drive an imagination-augmented exploration strategy to improve the model-free RL controller.

Remember the Past for Better Future: Memory-Augmented Offline RL

AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning

Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning

Online Reinforcement Learning with Passive Memory

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Sample Efficient Reinforcement Learning Using Graph-Based Memory Reconstruction.

Model-Based Offline Adaptive Policy Optimization with Episodic Memory

Reinforcement Learning with Fast and Forgetful Memory

Augmented Replay Memory in Reinforcement Learning With Continuous Control

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

Learning a World Model With Multitimescale Memory Augmentation

Understanding and Alleviating Memory Consumption in RLHF for LLMs

Ordered Memory.

On the Structural Memory of LLM Agents

Reinforcement Learning for Dynamic Memory Allocation

Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation

A Dual Memory Structure for Efficient Use of Replay Memory in Deep Reinforcement Learning

Two-Memory Reinforcement Learning

Revisiting Recurrent Reinforcement Learning with Memory Monoids

Model-Based Reinforcement Learning Via Imagination with Derived Memory.

Semantic HELM: A Human-Readable Memory for Reinforcement Learning