Abstract:In this work, we investigate the utilization of deep approximate policy iteration (DAPI) in estimating the optimal action-value function Q<sup>*</sup> within the context of reinforcement learning, employing rectified linear unit (ReLU) ResNet as the underlying framework. The iterative process of DAPI incorporates the minimax average Bellman error minimization principle. It employs ReLU ResNet to estimate the fixed point of the Bellman equation, which is aligned with the estimated greedy policy. Through error propagation, we derive nonasymptotic error bounds between Q<sup>*</sup> and the estimated Q function induced by the output greedy policy in DAPI. To effectively control the Bellman residual error, we address both the statistical and approximation errors associated with the α -mixing dependent data derived from Markov decision processes, using the techniques of empirical process and deep approximation theory, respectively. Furthermore, we present a novel generalization bound for ReLU ResNet in the presence of dependent data, as well as an approximation bound for ReLU ResNet within the Hölder class. Notably, this approximation bound contributes to a significant improvement in the dependence on the ambient dimension, transitioning from an exponential relationship to a polynomial one. The derived nonasymptotic error bounds explicitly depend on factors such as the sample size, the ambient dimension (in polynomial terms), and the width and depth of the neural networks. Consequently, these bounds serve as valuable theoretical guidelines for appropriately setting the hyperparameters, thereby enabling the achievement of the desired convergence rate during the training process of DAPI.

Adaptive Approximation Strategy to Reduce Approximation Error in Reinforcement Learning

Approximate Policy Iteration With Deep Minimax Average Bellman Error Minimization

Error Controlled Actor-Critic

Addressing Function Approximation Error in Actor-Critic Methods

Actor-Critic With Synthesis Loss for Solving Approximation Biases

Avoiding Confusion between Predictors and Inhibitors in Value Function Approximation

Regret Minimization for Partially Observable Deep Reinforcement Learning

Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback

Reducing Estimation Bias Via Triplet-Average Deep Deterministic Policy Gradient.

Computationally Efficient Imitation Learning Via K-Timestep Adaptive Action Approximation

Model predictive control-based value estimation for efficient reinforcement learning

Reinforcement Learning Algorithms with Function Approximation: Recent Advances and Applications

The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation

Controlling Estimation Error in Reinforcement Learning via Reinforced Operation

Understanding Deep Neural Function Approximation in Reinforcement Learning via $ε$-Greedy Exploration

Blending MPC & Value Function Approximation for Efficient Reinforcement Learning

Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learning

A Heuristic Dyna Optimizing Algorithm Using Approximate Model Representation

Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Reward

Data Efficient Deep Reinforcement Learning with Action-Ranked Temporal Difference Learning

Successively Pruned Q-Learning: Using Self Q-function to Reduce the Overestimation.