Abstract:For infinite-horizon average-cost criterion problems, there exist relatively few rigorous approximation and reinforcement learning results. In this paper, for such problems, we present several approximation and reinforcement learning results for Markov Decision Processes with standard Borel spaces. Toward this end, (i) we first provide a discretization based approximation method for fully observed Markov Decision Processes (MDPs) with continuous spaces under average cost criteria, and we provide error bounds for the approximations when the dynamics are only weakly continuous (for asymptotic convergence of errors as the grid sizes vanish) or Wasserstein continuous (with a rate in approximation as the grid sizes vanish) under certain ergodicity assumptions. In particular, we relax the total variation condition given in prior work to weak continuity as well as Wasserstein continuity conditions. (ii) We provide synchronous and asynchronous Q-learning algorithms for continuous spaces via quantization (where the quantized state is taken to be the actual state in corresponding Q-learning algorithms presented in the paper), and establish their convergence; for the former we utilize a span semi-norm approach and for the latter we use a direct contraction approach. (iii) We finally show that the convergence is to the optimal Q values of the finite approximate models constructed via quantization, which implies near optimality of the arrived solution. Our Q-learning convergence results and their convergence to near optimality are new for continuous spaces, and the proof method is new even for finite spaces, to our knowledge.

A Q-learning algorithm for Markov decision processes with continuous state spaces

Relative Q-Learning for Average-Reward Markov Decision Processes with Continuous States

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Q-Learning for Continuous State and Action MDPs under Average Cost Criteria

Gradient Q(σ, Λ): A Unified Algorithm with Function Approximation for Reinforcement Learning

Asynchronous value iteration for markov decision processes with continuous state spaces

Average Cost Optimal Control of Stochastic Systems Using Reinforcement Learning

Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimality

Approximate Kalman Filter Q-Learning for Continuous State-Space MDPs

Decentralised Q-Learning for Multi-Agent Markov Decision Processes with a Satisfiability Criterion

Minimax Optimal Q Learning with Nearest Neighbors

Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization

Online Adaptive Optimal Control Algorithm Based on Synchronous Integral Reinforcement Learning With Explorations

Online Markov decision processes with policy iteration

Regularized Q-Learning with Linear Function Approximation

Online Adaptive Optimization Algorithm for Semi-Markov Control Processes

Optimal tracking control of batch processes with time-invariant state delay: Adaptive Q-learning with two-dimensional state and control policy

Blackwell Online Learning for Markov Decision Processes

Continuous-time q-learning for mean-field control problems

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes

A Statistical Analysis of Polyak-Ruppert Averaged Q-learning