Abstract:Deep reinforcement learning (DRL) has shown remarkable capabilities in solving sequential decision-making problems. However, DRL requires extensive interactions with image-based environments. Existing methods have combined self-supervised learning or data augmentation to improve sample efficiency. While understanding the temporal information dynamics of the environment is important for effective learning, many methods do not consider these factors. To address the sample efficiency problem, we propose dynamics modeling with sequential temporal information (DynaSTI) that incorporates environmental dynamics and leverages the correlation among trajectories to improve sample efficiency. DynaSTI uses an effective learning strategy for state representation as an auxiliary task, using gated recurrent units to capture temporal information. It also integrates forward and inverse dynamics modeling in a hierarchical configuration, enhancing the learning of environmental dynamics compared to using each model separately. The hierarchical structure of DynaSTI enhances the stability of inverse dynamics modeling during training by using inputs derived from forward dynamics modeling, which focuses on feature extraction related to controllable state. This approach effectively filters out noisy information. Consequently, using denoised inputs from forward dynamics modeling results in improved stability when training inverse dynamics modeling, rather than using inputs directly from the encoder. We demonstrate the effectiveness of DynaSTI through experiments on the Atari game benchmark, limiting the environment interactions to 100k steps. Our extensive experiments confirm that DynaSTI significantly improves the sample efficiency of DRL, outperforming comparison methods in terms of statistically reliable metrics and nearing human-level performance.

DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning

Meta-Reinforcement Learning Algorithm Based on Reward and Dynamic Inference

Sample-efficient multi-agent reinforcement learning with masked reconstruction

MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning

Meta-Reinforcement Learning with Dynamic Adaptiveness Distillation

DynaSTI: Dynamics Modeling with Sequential Temporal Information for Reinforcement Learning in Atari

Sample-Efficient Reinforcement Learning Based on Dynamics Models via Meta-policy Optimization

Temporal Difference Models: Model-Free Deep RL for Model-Based Control

Model-Based Reinforcement Learning via Meta-Policy Optimization

Simplified Temporal Consistency Reinforcement Learning

Episodic Reinforcement Learning with Expanded State-reward Space

Dyna-style Model-based reinforcement learning with Model-Free Policy Optimization

Dream to Adapt: Meta Reinforcement Learning by Latent Context Imagination and MDP Imagination

NoRML: No-Reward Meta Learning

MELD: Meta-Reinforcement Learning from Images via Latent State Models

Doubly Robust Augmented Transfer for Meta-Reinforcement Learning.

Model-based Adversarial Meta-Reinforcement Learning

Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL

Masked and Inverse Dynamics Modeling for Data-Efficient Reinforcement Learning

A Digital Twin Framework for Reinforcement Learning with Real-Time Self-Improvement via Human Assistive Teleoperation

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning