Abstract:Deep reinforcement learning for high dimensional, hierarchical control tasks usually requires the use of complex neural networks as functional approximators, which can lead to inefficiency, instability and even divergence in the training process. Here, we introduce stacked deep Q learning (SDQL), a flexible modularized deep reinforcement learning architecture, that can enable finding of optimal control policy of control tasks consisting of multiple linear stages in a stable and efficient way. SDQL exploits the linear stage structure by approximating the Q function via a collection of deep Q sub-networks stacking along an axis marking the stage-wise progress of the whole task. By back-propagating the learned state values from later stages to earlier stages, all sub-networks co-adapt to maximize the total reward of the whole task, although each sub-network is responsible for learning optimal control policy for its own stage. This modularized architecture offers considerable flexibility in terms of environment and policy modeling, as it allows choices of different state spaces, action spaces, reward structures, and Q networks for each stage, Further, the backward stage-wise training procedure of SDQL can offers additional transparency, stability, and flexibility to the training process, thus facilitating model fine-tuning and hyper-parameter search. We demonstrate that SDQL is capable of learning competitive strategies for problems with characteristics of high-dimensional state space, heterogeneous action space(both discrete and continuous), multiple scales, and sparse and delayed rewards.

Deep Q-Learning with Low Switching Cost

A Benchmark for Low-Switching-Cost Reinforcement Learning

Switch-Based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning.

To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning

Deep Q-Learning with Prioritized Sampling.

Small cell switch policy: A reinforcement learning approach

Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning

Deep Reinforcement Learning with Spiking Q-learning

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation

Finite-Time Analysis of Asynchronous Q-Learning Under Diminishing Step-Size From Control-Theoretic View

DQN with model-based exploration: efficient learning on environments with sparse rewards

Reducing the Deployment-Time Inference Control Costs of Deep Reinforcement Learning Agents via an Asymmetric Architecture

Fuzzy Optimal Control for a Class of Discrete-Time Switched Nonlinear Systems

A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Cost

State Distribution-Aware Sampling for Deep Q-Learning

Deep Reinforcement Learning with Double Q-Learning

Deep Reinforcement Learning: from Q-Learning to Deep Q-Learning.

A Deep Reinforcement Learning Architecture for Multi-stage Optimal Control

Finite-Time Analysis of Minimax Q-Learning for Two-Player Zero-Sum Markov Games: Switching System Approach

Using Deep Q-Learning to Control Optimization Hyperparameters

Learning to Switch Among Agents in a Team via 2-Layer Markov Decision Processes