Abstract:Visual reinforcement learning is important in various practical applications, such as video games, robotic manipulation, and autonomous navigation. However, a major challenge in visual reinforcement learning is the generalization to unseen environments, that is, how agents manage environments with previously unseen backgrounds. This issue is triggered mainly by the high unpredictability inherent in high-dimensional observation space. To deal with this problem, techniques including domain randomization and data augmentation have been explored; nevertheless, these methods still cannot attain a satisfactory result. This paper proposes a new method named Internal States Simulation Auxiliary (ISSA), which uses internal states to improve generalization in visual reinforcement learning tasks. Our method contains two agents, a teacher agent and a student agent: the teacher agent has the ability to directly access the environment's internal states and is used to facilitate the student agent's training; the student agent receives initial guidance from the teacher agent and subsequently continues to learn independently. From another perspective, our method can be divided into two phases, the transfer learning phase and traditional visual reinforcement learning phase. In the first phase, the teacher agent interacts with environments and imparts knowledge to the vision-based student agent. With the guidance of the teacher agent, the student agent is able to discover more effective visual representations that address the high unpredictability of high-dimensional observation space. In the next phase, the student agent autonomously learns from the visual information in the environment, and ultimately, it becomes a vision-based reinforcement learning agent with enhanced generalization. The effectiveness of our method is evaluated using the DMControl Generalization Benchmark and the DrawerWorld with texture distortions. Preliminary results indicate that our method significantly improves generalization ability and performance in complex continuous control tasks.

Internal Logical Induction for Pixel-Symbolic Reinforcement Learning

Incorporating Relational Background Knowledge into Reinforcement Learning via Differentiable Inductive Logic Programming

Deep Inductive Logic Programming meets Reinforcement Learning

Imperative Learning: A Self-supervised Neural-Symbolic Learning Framework for Robot Autonomy

Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search

RLIF: Interactive Imitation Learning as Reinforcement Learning

RL-I2IT: Image-to-Image Translation with Deep Reinforcement Learning

Blending Imitation and Reinforcement Learning for Robust Policy Improvement

Learning Controllable Elements Oriented Representations for Reinforcement Learning

Achieving efficient interpretability of reinforcement learning via policy distillation and selective input gradient regularization

Generalization Enhancement of Visual Reinforcement Learning through Internal States

Following Instructions by Imagining and Reaching Visual Goals

Relate to Predict: Towards Task-Independent Knowledge Representations for Reinforcement Learning

Interpretable Reinforcement Learning Inspired by Piaget's Theory of Cognitive Development

ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI

Internally Rewarded Reinforcement Learning

Robust Visual Imitation Learning with Inverse Dynamics Representations

LIIR: Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning.

Combining RL and IL using a dynamic, performance-based modulation over learning signals and its application to local planning

BlendRL: A Framework for Merging Symbolic and Neural Policy Learning

EvIL: Evolution Strategies for Generalisable Imitation Learning