Abstract:For infinite-horizon average-cost criterion problems, there exist relatively few rigorous approximation and reinforcement learning results. In this paper, for such problems, we present several approximation and reinforcement learning results for Markov Decision Processes with standard Borel spaces. Toward this end, (i) we first provide a discretization based approximation method for fully observed Markov Decision Processes (MDPs) with continuous spaces under average cost criteria, and we provide error bounds for the approximations when the dynamics are only weakly continuous (for asymptotic convergence of errors as the grid sizes vanish) or Wasserstein continuous (with a rate in approximation as the grid sizes vanish) under certain ergodicity assumptions. In particular, we relax the total variation condition given in prior work to weak continuity as well as Wasserstein continuity conditions. (ii) We provide synchronous and asynchronous Q-learning algorithms for continuous spaces via quantization (where the quantized state is taken to be the actual state in corresponding Q-learning algorithms presented in the paper), and establish their convergence; for the former we utilize a span semi-norm approach and for the latter we use a direct contraction approach. (iii) We finally show that the convergence is to the optimal Q values of the finite approximate models constructed via quantization, which implies near optimality of the arrived solution. Our Q-learning convergence results and their convergence to near optimality are new for continuous spaces, and the proof method is new even for finite spaces, to our knowledge.

Relative Q-Learning for Average-Reward Markov Decision Processes with Continuous States

A Q-learning algorithm for Markov decision processes with continuous state spaces

Q-Learning for Continuous State and Action MDPs under Average Cost Criteria

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes

Sharper Model-free Reinforcement Learning for Average-reward Markov Decision Processes

Study on an Average Reward Reinforcement Learning Algorithm

Model-Free Robust Average-Reward Reinforcement Learning

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation

An immediate-return reinforcement learning for the atypical Markov decision processes

Asynchronous Stochastic Approximation and Average-Reward Reinforcement Learning

A Statistical Analysis of Polyak-Ruppert Averaged Q-learning

Asynchronous value iteration for markov decision processes with continuous state spaces

Average Reward Reinforcement Learning For Semi-Markov Decision Processes

Decentralised Q-Learning for Multi-Agent Markov Decision Processes with a Satisfiability Criterion

Logarithmic regret bounds for continuous-time average-reward Markov decision processes

A Structure-aware Online Learning Algorithm for Markov Decision Processes

Reinforcement learning algorithms for semi-Markov decision processes with average reward

Regularized Q-Learning with Linear Function Approximation

Optimal Sample Complexity for Average Reward Markov Decision Processes