Abstract:How to improve the ability of scene representation is a key issue in vision-oriented decision-making applications, and current approaches usually learn task-relevant state representations within visual reinforcement learning to address this problem. While prior work typically introduces one-step behavioral similarity metrics with elements (e.g., rewards and actions) to extract task-relevant state information from observations, they often ignore the inherent dynamics relationships among the elements that are essential for learning accurate representations, which further impedes the discrimination of short-term similar task/behavior information in long-term dynamics transitions. To alleviate this problem, we propose an intrinsic dynamics-driven representation learning method with sequence models in visual reinforcement learning, namely DSR. Concretely, DSR optimizes the parameterized encoder by the state-transition dynamics of the underlying system, which prompts the latent encoding information to satisfy the state-transition process and then the state space and the noise space can be distinguished. In the implementation and to further improve the representation ability of DSR on encoding similar tasks, sequential elements' frequency domain and multi-step prediction are adopted for sequentially modeling the inherent dynamics. Finally, experimental results show that DSR has achieved significant performance improvements in the visual Distracting DMControl control tasks, especially with an average of 78.9\% over the backbone baseline. Further results indicate that it also achieves the best performances in real-world autonomous driving applications on the CARLA simulator. Moreover, qualitative analysis results validate that our method possesses the superior ability to learn generalizable scene representations on visual tasks. The source code is available at <a class="link-external link-https" href="https://github.com/DMU-XMU/DSR" rel="external noopener nofollow">this https URL</a>.

Discovery and learning of models with predictive state representations for dynamical systems without reset

Survey of Predictive State Representations

Survey on predictive representations of state

Model-Based Robot Learning Control with Uncertainty Directed Exploration

Extended State Space Predictive Control for a Class of Nonlinear Systems

Fast and Stable Learning of Dynamical Systems Based on Extreme Learning Machine.

Provably Efficient UCB-type Algorithms For Learning Predictive State Representations

Bridging State and History Representations: Understanding Self-Predictive RL

Hilbert Space Embeddings of Predictive State Representations

DISCOVER: Deep identification of symbolically concise open-form PDEs via enhanced reinforcement-learning

HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context

Discovering Sparse Interpretable Dynamics from Partial Observations

Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithms

Predictive-State Decoders: Encoding the Future into Recurrent Networks

Learning Deep Dynamical Systems using Stable Neural ODEs

ODE-based Recurrent Model-free Reinforcement Learning for POMDPs

Learning State Representations via Retracing in Reinforcement Learning

Towards Learning Controllable Representations of Physical Systems

Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning

Intrinsic Dynamics-Driven Generalizable Scene Representations for Vision-Oriented Decision-Making Applications

State Representation Learning with Adjacent State Consistency Loss for Deep Reinforcement Learning.