Abstract:Visual reinforcement learning is important in various practical applications, such as video games, robotic manipulation, and autonomous navigation. However, a major challenge in visual reinforcement learning is the generalization to unseen environments, that is, how agents manage environments with previously unseen backgrounds. This issue is triggered mainly by the high unpredictability inherent in high-dimensional observation space. To deal with this problem, techniques including domain randomization and data augmentation have been explored; nevertheless, these methods still cannot attain a satisfactory result. This paper proposes a new method named Internal States Simulation Auxiliary (ISSA), which uses internal states to improve generalization in visual reinforcement learning tasks. Our method contains two agents, a teacher agent and a student agent: the teacher agent has the ability to directly access the environment's internal states and is used to facilitate the student agent's training; the student agent receives initial guidance from the teacher agent and subsequently continues to learn independently. From another perspective, our method can be divided into two phases, the transfer learning phase and traditional visual reinforcement learning phase. In the first phase, the teacher agent interacts with environments and imparts knowledge to the vision-based student agent. With the guidance of the teacher agent, the student agent is able to discover more effective visual representations that address the high unpredictability of high-dimensional observation space. In the next phase, the student agent autonomously learns from the visual information in the environment, and ultimately, it becomes a vision-based reinforcement learning agent with enhanced generalization. The effectiveness of our method is evaluated using the DMControl Generalization Benchmark and the DrawerWorld with texture distortions. Preliminary results indicate that our method significantly improves generalization ability and performance in complex continuous control tasks.

Enhancing Visual Reinforcement Learning with State-Action Representation

Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning

Effective Representation Learning is More Effective in Reinforcement Learning Than You Think

State Representation Learning for Effective Deep Reinforcement Learning.

TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning

State Representation Learning with Adjacent State Consistency Loss for Deep Reinforcement Learning.

For SALE: State-Action Representation Learning for Deep Reinforcement Learning

Representation learning for continuous action spaces is beneficial for efficient policy learning

Learning explainable task-relevant state representation for model-free deep reinforcement learning

Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning

A stable data-augmented reinforcement learning method with ensemble exploration and exploitation

Task-Induced Representation Learning

STACoRe: Spatio-temporal and Action-Based Contrastive Representations for Reinforcement Learning in Atari

When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?

Intrinsic Dynamics-Driven Generalizable Scene Representations for Vision-Oriented Decision-Making Applications

Stabilizing Visual Reinforcement Learning Via Asymmetric Interactive Cooperation

Improving Reinforcement Learning Efficiency with Auxiliary Tasks in Non-Visual Environments: A Comparison

Learning Controllable Elements Oriented Representations for Reinforcement Learning

Learning State Representations via Retracing in Reinforcement Learning

State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping

Generalization Enhancement of Visual Reinforcement Learning through Internal States