Abstract:Model-based reinforcement learning (RL) is regarded as a promising approach to tackle the challenges that hinder model-free RL. The success of model-based RL hinges critically on the quality of the predicted dynamic models. However, for many real-world tasks involving high-dimensional state spaces, current dynamics prediction models show poor performance in long-term prediction. To that end, we propose a novel two-branch neural network architecture with multi-timescale memory augmentation to handle long-term and short-term memory differently. Specifically, we follow previous works to introduce a recurrent neural network architecture to encode history observation sequences into latent space, characterizing the long-term memory of agents. Different from previous works, we view the most recent observations as the short-term memory of agents and employ them to directly reconstruct the next frame to avoid compounding error. This is achieved by introducing a self-supervised optical flow prediction structure to model the action-conditional feature transformation at pixel level. The reconstructed observation is finally augmented by the long-term memory to ensure semantic consistency. Experimental results show that our approach is able to generate visually-realistic long-term predictions in DeepMind maze navigation games, and outperforms the prevalent state-of-the-art methods in prediction accuracy by a large margin. Furthermore, we also evaluate the usefulness of our world model by using the predicted frames to drive an imagination-augmented exploration strategy to improve the model-free RL controller.

Model-Based Reinforcement Learning Via Imagination with Derived Memory.

Model-Based Off-Policy Deep Reinforcement Learning with Model-Embedding

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Learning a World Model With Multitimescale Memory Augmentation

Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space

AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning

Model-Based Reinforcement Learning with Isolated Imaginations

Robust Memory Augmentation by Constrained Latent Imagination

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning

Sample Efficient Reinforcement Learning Using Graph-Based Memory Reconstruction.

Mastering Memory Tasks with World Models

Model Embedding Model-Based Reinforcement Learning

Mnemonic Dictionary Learning for Intrinsic Motivation in Reinforcement Learning

Influence-aware Memory Architectures for Deep Reinforcement Learning

Learning Latent Dynamic Robust Representations for World Models

Open-World Reinforcement Learning over Long Short-Term Imagination

Model-based reinforcement learning with dimension reduction

Integrated model of cerebellal supervised learning and basal ganglia's reinforcement learning for mobile robot behavioral decision-making

Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning

Dream to Control: Learning Behaviors by Latent Imagination