Abstract:Visual reinforcement learning is important in various practical applications, such as video games, robotic manipulation, and autonomous navigation. However, a major challenge in visual reinforcement learning is the generalization to unseen environments, that is, how agents manage environments with previously unseen backgrounds. This issue is triggered mainly by the high unpredictability inherent in high-dimensional observation space. To deal with this problem, techniques including domain randomization and data augmentation have been explored; nevertheless, these methods still cannot attain a satisfactory result. This paper proposes a new method named Internal States Simulation Auxiliary (ISSA), which uses internal states to improve generalization in visual reinforcement learning tasks. Our method contains two agents, a teacher agent and a student agent: the teacher agent has the ability to directly access the environment's internal states and is used to facilitate the student agent's training; the student agent receives initial guidance from the teacher agent and subsequently continues to learn independently. From another perspective, our method can be divided into two phases, the transfer learning phase and traditional visual reinforcement learning phase. In the first phase, the teacher agent interacts with environments and imparts knowledge to the vision-based student agent. With the guidance of the teacher agent, the student agent is able to discover more effective visual representations that address the high unpredictability of high-dimensional observation space. In the next phase, the student agent autonomously learns from the visual information in the environment, and ultimately, it becomes a vision-based reinforcement learning agent with enhanced generalization. The effectiveness of our method is evaluated using the DMControl Generalization Benchmark and the DrawerWorld with texture distortions. Preliminary results indicate that our method significantly improves generalization ability and performance in complex continuous control tasks.

Exploiting Generalization in the Subspaces for Faster Model-Based Reinforcement Learning

Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning

The Benefits of Model-Based Generalization in Reinforcement Learning

Learning Parsimonious Dynamics for Generalization in Reinforcement Learning

Generalization Enhancement of Visual Reinforcement Learning through Internal States

Model-Based Reinforcement Learning via Meta-Policy Optimization

MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure

Stronger Generalization Guarantees for Robot Learning by Combining Generative Models and Real-World Data

Learning Latent Dynamic Robust Representations for World Models

Focus On What Matters: Separated Models For Visual-Based RL Generalization

Enhance Generality by Model-based Reinforcement Learning and Domain Randomization

Representation learning for continuous action spaces is beneficial for efficient policy learning

Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation

TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matching

Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning

Model-Based Off-Policy Deep Reinforcement Learning with Model-Embedding

Goal-Space Planning with Subgoal Models

Transfer Reinforcement Learning in Heterogeneous Action Spaces using Subgoal Mapping

Identifying Policy Gradient Subspaces

An Efficient Model-Based Approach on Learning Agile Motor Skills without Reinforcement

Policy-shaped prediction: avoiding distractions in model-based reinforcement learning