Abstract:We study the problem of average-reward Markov decision processes (AMDPs) and develop novel first-order methods with strong theoretical guarantees for both policy evaluation and optimization. Existing on-policy evaluation methods suffer from sub-optimal convergence rates as well as failure in handling insufficiently random policies, e.g., deterministic policies, for lack of exploration. To remedy these issues, we develop a novel variance-reduced temporal difference (VRTD) method with linear function approximation for randomized policies along with sharp convergence guarantees, and an exploratory variance-reduced temporal difference (EVRTD) method for insufficiently random policies with comparable convergence guarantees. We further establish linear convergence rate on the bias of policy evaluation, which is essential for improving the overall sample complexity of policy optimization. On the other hand, compared with intensive research interest in finite sample analysis of policy gradient methods for discounted MDPs, existing studies on policy gradient methods for AMDPs mostly focus on regret bounds under restrictive assumptions on the underlying Markov processes (see, e.g., Abbasi-Yadkori et al., 2019), and they often lack guarantees on the overall sample complexities. Towards this end, we develop an average-reward variant of the stochastic policy mirror descent (SPMD) (Lan, 2022). We establish the first $\widetilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity for solving AMDPs with policy gradient method under both the generative model (with unichain assumption) and Markovian noise model (with ergodic assumption). This bound can be further improved to $\widetilde{\mathcal{O}}(\epsilon^{-1})$ for solving regularized AMDPs. Our theoretical advantages are corroborated by numerical experiments.

Simulation-Based optimization of singularly perturbed markov reward processes with states aggregation

A State Aggregation Approach to Singularly Perturbed Markov Reward Processes

Simulation Optimization Algorithm for SMDPs with Parameterized Randomized Stationary Policies

Asynchronous Stochastic Approximation and Average-Reward Reinforcement Learning

Online Reinforcement Learning via Posterior Sampling of Policy

Average Reward Reinforcement Learning For Semi-Markov Decision Processes

Sharper Model-free Reinforcement Learning for Average-reward Markov Decision Processes

Two Time-Scale Gradient Approximation Algorithm For Adaptive Markov Reward Processes

Relative Q-Learning for Average-Reward Markov Decision Processes with Continuous States

A Simulation Optimization Algorithm for CTMDPs Based on Randomized Stationary Policies

Recursive Approaches for Single Sample Path Based Markov Reward Processes

Reinforcement learning algorithms for semi-Markov decision processes with average reward

RVI reinforcement learning for semi-Markov decision processes with average reward

Potential Based Optimization Algorithm Of Constrained Markov Decision Processes

A unified approach for semi-Markov decision processes with discounted and average reward criteria

Optimal Sample Complexity for Average Reward Markov Decision Processes

Parallel Optimization for Markov Control Processes Based on Performance Potentials Simulation

Error bounds of optimization algorithms for semi-Markov decision processes

Asynchronous value iteration for markov decision processes with continuous state spaces

Stochastic first-order methods for average-reward Markov decision processes

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes