Abstract:We consider the problem of solving robust Markov decision process (MDP), which involves a set of discounted, finite state, finite action space MDPs with uncertain transition kernels. The goal of planning is to find a robust policy that optimizes the worst-case values against the transition uncertainties, and thus encompasses the standard MDP planning as a special case. For $(\mathbf{s},\mathbf{a})$-rectangular uncertainty sets, we develop a policy-based first-order method, namely the robust policy mirror descent (RPMD), and establish an $\mathcal{O}(\log(1/\epsilon))$ and $\mathcal{O}(1/\epsilon)$ iteration complexity for finding an $\epsilon$-optimal policy, with two increasing-stepsize schemes. The prior convergence of RPMD is applicable to any Bregman divergence, provided the policy space has bounded radius measured by the divergence when centering at the initial policy. Moreover, when the Bregman divergence corresponds to the squared euclidean distance, we establish an $\mathcal{O}(\max \{1/\epsilon, 1/(\eta \epsilon^2)\})$ complexity of RPMD with any constant stepsize $\eta$. For a general class of Bregman divergences, a similar complexity is also established for RPMD with constant stepsizes, provided the uncertainty set satisfies the relative strong convexity. We further develop a stochastic variant, named SRPMD, when the first-order information is only available through online interactions with the nominal environment. For general Bregman divergences, we establish an $\mathcal{O}(1/\epsilon^2)$ and $\mathcal{O}(1/\epsilon^3)$ sample complexity with two increasing-stepsize schemes. For the euclidean Bregman divergence, we establish an $\mathcal{O}(1/\epsilon^3)$ sample complexity with constant stepsizes. To the best of our knowledge, all the aforementioned results appear to be new for policy-based first-order methods applied to the robust MDP problem.

Finding Near Optimal Policies via Reducive Regularization in Markov Decision Processes

Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPs

A Regularized Approach to Sparse Optimal Policy in Reinforcement Learning.

Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systems

Efficient Policy Iteration for Robust Markov Decision Processes via Regularization

Stochastic Cubic-Regularized Policy Gradient Method

Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action

A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes

Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

Robust Average-Reward Markov Decision Processes

Optimistic Planning by Regularized Dynamic Programming

Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form

Policy Gradient for Robust Markov Decision Processes

Beyond discounted returns: Robust Markov decision processes with average and Blackwell optimality

Safe Reinforcement Learning in Constrained Markov Decision Processes

The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning

A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processes

First-order Policy Optimization for Robust Markov Decision Process

Double Duality: Variational Primal-Dual Policy Optimization for Constrained Reinforcement Learning

Finding Optimal Memoryless Policies of POMDPs under the Expected Average Reward Criterion