Abstract:Deep reinforcement learning (DRL) gives the promise that an agent learns good policy from high-dimensional information, whereas representation learning removes irrelevant and redundant information and retains pertinent information. In this work, we demonstrate that the learned representation of the $Q$-network and its target $Q$-network should, in theory, satisfy a favorable distinguishable representation property. Specifically, there exists an upper bound on the representation similarity of the value functions of two adjacent time steps in a typical DRL setting. However, through illustrative experiments, we show that the learned DRL agent may violate this property and lead to a sub-optimal policy. Therefore, we propose a simple yet effective regularizer called Policy Evaluation with Easy Regularization on Representation (PEER), which aims to maintain the distinguishable representation property via explicit regularization on internal representations. And we provide the convergence rate guarantee of PEER. Implementing PEER requires only one line of code. Our experiments demonstrate that incorporating PEER into DRL can significantly improve performance and sample efficiency. Comprehensive experiments show that PEER achieves state-of-the-art performance on all 4 environments on PyBullet, 9 out of 12 tasks on DMControl, and 19 out of 26 games on Atari. To the best of our knowledge, PEER is the first work to study the inherent representation property of Q-network and its target. Our code is available at <a class="link-external link-https" href="https://sites.google.com/view/peer-cvpr2023/" rel="external noopener nofollow">this https URL</a>.

A State Representation Dueling Network for Deep Reinforcement Learning

Dueling Network Architecture for Multi-Agent Deep Deterministic Policy Gradient

State Representation Learning for Effective Deep Reinforcement Learning.

State Representation Learning with Adjacent State Consistency Loss for Deep Reinforcement Learning.

Double Sparse Deep Reinforcement Learning Via Multilayer Sparse Coding and Nonconvex Regularized Pruning

A Dueling Deep Recurrent Q -Network Framework for Dynamic Multichannel Access in Heterogeneous Wireless Networks

Based on Doubly Decoupled Reinforced Network

State of the Art Control of Atari Games Using Shallow Reinforcement Learning

Research on Autonomous Obstacle Avoidance and Target Tracking of UAV Based on Improved Dueling DQN Algorithm

Reinforcement Learning with Decoupled State Representation for Robot Manipulations

Deep Q-learning From Demonstrations

Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learning

End-to-End Autonomous Driving Through Dueling Double Deep Q-Network

Task Offloading Via Prioritized Experience-based Double Dueling DQN in Edge-assisted IIoT

Multi-Experience Pool Local State Parallel Q-Network.

M$^2$DQN: A Robust Method for Accelerating Deep Q-learning Network

Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples

DeRL: Coupling Decomposition in Action Space for Reinforcement Learning Task

Reducing overestimation in value mixing for cooperative deep multi-agent reinforcement learning

Deep Reinforcement Learning with Decorrelation

Study on the Path Planning Algorithm Based on Dueling Deep Q Network