Abstract:In reinforcement learning (RL), learning directly from pixels, is commonly known as vision-based RL. Effective state representations are crucial for high performance in vision-based RL. However, in order to learn effective state representations, most current vision-based RL methods based on contrastive unsupervised learning use auxiliary tasks similar to those in computer vision, which does not guarantee the effective information interaction between representation learning and RL. To learn more efficient states, we propose a simple and effective vision-based RL method. It leverages the representations acquired through contrastive learning by the Teacher Encoder and the Student Encoder to collaboratively estimate the Q-function. This cooperative process utilizes the TD error to steer updates to the Teacher Encoder, thereby ensuring effective information exchange between representation learning and RL. We refer to this approach as Reinforcement Learning with Teacher-Student Collaboration (RLTSC). RLTSC incorporates recent advancements in contrastive unsupervised learning, endowing it with potent representation learning capabilities. It provides a robust estimate of the Q-function with minimal variance and effectively guides the Teacher Ecoder to update and acquire a more efficient representation. RLTSC substantially enhances data efficiency in vision-based RL, surpassing state-of-the-art methods on various continuous and discrete control benchmarks. Remarkably, RLTSC even outperforms RL methods based on physical state features in terms of data efficiency for continuous control benchmarks. This may enlighten us: effective representation learning is more effective in reinforcement learning than you think!

State Representation Learning for Effective Deep Reinforcement Learning.

State Representation Learning with Adjacent State Consistency Loss for Deep Reinforcement Learning.

S2RL: DoWe Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Domain Adaptive State Representation Alignment for Reinforcement Learning

Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning

A State Representation Dueling Network for Deep Reinforcement Learning

Learning explainable task-relevant state representation for model-free deep reinforcement learning

Effective Representation Learning is More Effective in Reinforcement Learning Than You Think

Enhancing Visual Reinforcement Learning with State-Action Representation

State of the Art Control of Atari Games Using Shallow Reinforcement Learning

Episodic Reinforcement Learning with Expanded State-reward Space

Learning Controllable Elements Oriented Representations for Reinforcement Learning

Neural Episodic Control with State Abstraction

Bootstrapped Representations in Reinforcement Learning

Unsupervised State Representation Learning in Atari

Double Sparse Deep Reinforcement Learning Via Multilayer Sparse Coding and Nonconvex Regularized Pruning

LLM-Empowered State Representation for Reinforcement Learning

Improving Deep Reinforcement Learning with Mirror Loss

For SALE: State-Action Representation Learning for Deep Reinforcement Learning

Learning to Identify Critical States for Reinforcement Learning from Videos