Abstract:We study infinite-horizon average-reward Markov decision processes (AMDPs) in the context of general function approximation. Specifically, we propose a novel algorithmic framework named Local-fitted Optimization with OPtimism (LOOP), which incorporates both model-based and value-based incarnations. In particular, LOOP features a novel construction of confidence sets and a low-switching policy updating scheme, which are tailored to the average-reward and function approximation setting. Moreover, for AMDPs, we propose a novel complexity measure -- average-reward generalized eluder coefficient (AGEC) -- which captures the challenge of exploration in AMDPs with general function approximation. Such a complexity measure encompasses almost all previously known tractable AMDP models, such as linear AMDPs and linear mixture AMDPs, and also includes newly identified cases such as kernel AMDPs and AMDPs with Bellman eluder dimensions. Using AGEC, we prove that LOOP achieves a sublinear $\tilde{\mathcal{O}}(\mathrm{poly}(d, \mathrm{sp}(V^*)) \sqrt{T\beta} )$ regret, where $d$ and $\beta$ correspond to AGEC and log-covering number of the hypothesis class respectively, $\mathrm{sp}(V^*)$ is the span of the optimal state bias function, $T$ denotes the number of steps, and $\tilde{\mathcal{O}} (\cdot) $ omits logarithmic factors. When specialized to concrete AMDP models, our regret bounds are comparable to those established by the existing algorithms designed specifically for these special cases. To the best of our knowledge, this paper presents the first comprehensive theoretical framework capable of handling nearly all AMDPs.

Optimizing Local Satisfaction of Long-Run Average Objectives in Markov Decision Processes

Efficient Strategy Iteration for Mean Payoff in Markov Decision Processes

On Average Optimality for Non-Stationary Markov Decision Processes in Borel Spaces

Beyond discounted returns: Robust Markov decision processes with average and Blackwell optimality

Optimal models with maximizing probability of first achieving target value in the preceding stages

Bi-Objective Lexicographic Optimization in Markov Decision Processes with Related Objectives

Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action

Finding Optimal Memoryless Policies of POMDPs under the Expected Average Reward Criterion

Finding Optimal Observation-Based Policies for Constrained POMDPs under the Expected Average Reward Criterion

Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance

Narrowing the Gap between Adversarial and Stochastic MDPs via Policy Optimization

A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processes

MDP Geometry, Normalization and Reward Balancing Solvers

Global Algorithms for Mean-Variance Optimization in Markov Decision Processes

Beyond Average Return in Markov Decision Processes

Robust Average-Reward Markov Decision Processes

Risk-averse optimization of total rewards in Markovian models using deviation measures

Long-Term Resource Allocation Fairness in Average Markov Decision Process (AMDP) Environment

A Structure-aware Online Learning Algorithm for Markov Decision Processes

Reachability and Safety Objectives in Markov Decision Processes on Long but Finite Horizons

Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation