Abstract:We study the problem of average-reward Markov decision processes (AMDPs) and develop novel first-order methods with strong theoretical guarantees for both policy evaluation and optimization. Existing on-policy evaluation methods suffer from sub-optimal convergence rates as well as failure in handling insufficiently random policies, e.g., deterministic policies, for lack of exploration. To remedy these issues, we develop a novel variance-reduced temporal difference (VRTD) method with linear function approximation for randomized policies along with sharp convergence guarantees, and an exploratory variance-reduced temporal difference (EVRTD) method for insufficiently random policies with comparable convergence guarantees. We further establish linear convergence rate on the bias of policy evaluation, which is essential for improving the overall sample complexity of policy optimization. On the other hand, compared with intensive research interest in finite sample analysis of policy gradient methods for discounted MDPs, existing studies on policy gradient methods for AMDPs mostly focus on regret bounds under restrictive assumptions on the underlying Markov processes (see, e.g., Abbasi-Yadkori et al., 2019), and they often lack guarantees on the overall sample complexities. Towards this end, we develop an average-reward variant of the stochastic policy mirror descent (SPMD) (Lan, 2022). We establish the first $\widetilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity for solving AMDPs with policy gradient method under both the generative model (with unichain assumption) and Markovian noise model (with ergodic assumption). This bound can be further improved to $\widetilde{\mathcal{O}}(\epsilon^{-1})$ for solving regularized AMDPs. Our theoretical advantages are corroborated by numerical experiments.

Long-Term Resource Allocation Fairness in Average Markov Decision Process (AMDP) Environment

Sequential Fair Resource Allocation under a Markov Decision Process Framework

Fair Resource Allocation in Weakly Coupled Markov Decision Processes

Fairness over time in dynamic resource allocation with an application in healthcare

Towards Return Parity in Markov Decision Processes

Achieving Fairness in Multi-Agent Markov Decision Processes Using Reinforcement Learning

Stochastic first-order methods for average-reward Markov decision processes

Enabling Long-term Fairness in Dynamic Resource Allocation

Resource Allocation in Multi-channel Multi-user Relay System with Fairness Constraints

Online Resource Sharing via Dynamic Max-Min Fairness: Efficiency, Robustness and Non-Stationarity

Fair Graphical Resource Allocation with Matching-Induced Utilities

The Existence and Efficiency of PMMS Allocations

Achieving Maximin Share and EFX/EF1 Guarantees Simultaneously

Long-Term Fairness in Sequential Multi-Agent Selection with Positive Reinforcement

Possible Fairness for Allocating Indivisible Resources

Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systems

Resource Allocation Among Agents with MDP-Induced Preferences

Keeping the Harmony Between Neighbors: Local Fairness in Graph Fair Division

Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation

Efficient Resource Allocation with Fairness Constraints in Restless Multi-Armed Bandits

Online Resource Allocation in Episodic Markov Decision Processes