Abstract:This paper is concerned with complex reinforcement learning tasks whose observations are difficult to characterize as appropriate inputs for policy mapping. The representation learning technique is leveraged to extract the features from the observations for optimal action generation. In literature, the action vector, which consists of actions in each dimension, is usually learned from the full features. However, we find empirically that different actions may be only highly related to small part of the features and weakly depend on the rest. Therefore, this unified learning strategy may lead to performance degradation, and a separate learning method is motivated. In this paper, we propose a novel method called Decoupled Reinforcement Learning (DeRL) that decomposes action space by replacing the policy network with decoupled sub-policy group. To cater to all types of tasks where agents. actions in different dimensions can be either weakly correlated or strongly correlated, the Bidirectional Recurrent Neural Network (Bi-RNN) is added as an essential component to capture further shared features for generating more accurate action. In this framework, the decoupled policy network maintains joint representations required by the decision of all actions in different dimensions while decreasing preference in the learning process. In addition, we give a theoretical analysis of DeRL from the perspective of information theory, which shows the difference in information loss between DeRL and others. The performance of the proposed method has been verified by contrastive experiments on 12 tasks, including Mujoco, Atari, and other popular environments.

Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Network

Policy Return: A New Method for Reducing the Number of Experimental Trials in Deep Reinforcement Learning

A Model-Based Exploration Policy in Deep Q-Network

Dueling Network Architecture for Multi-Agent Deep Deterministic Policy Gradient

Deep Reinforcement Learning Using Least‐squares Truncated Temporal‐difference

A Q-based Policy Gradient Optimization Approach for Doudizhu

Data Efficient Deep Reinforcement Learning with Action-Ranked Temporal Difference Learning

Integrated Double Estimator Architecture for Reinforcement Learning

Combing Policy Evaluation and Policy Improvement in a Unified F-Divergence Framework.

Approximate Policy-Based Accelerated Deep Reinforcement Learning.

Exponential Moving Averaged Q-Network for DDPG

Non-crossing quantile regression for deep reinforcement learning

Dynamic Reward-Based Dueling Deep Dyna-Q: Robust Policy Learning in Noisy Environments

Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning

M$^2$DQN: A Robust Method for Accelerating Deep Q-learning Network

Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

Generalization and Regularization in DQN

Understanding the Synergies between Quality-Diversity and Deep Reinforcement Learning

Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space

DeRL: Coupling Decomposition in Action Space for Reinforcement Learning Task

Deep Reinforcement Learning with Double Q-Learning