Abstract:We consider the problem of solving robust Markov decision process (MDP), which involves a set of discounted, finite state, finite action space MDPs with uncertain transition kernels. The goal of planning is to find a robust policy that optimizes the worst-case values against the transition uncertainties, and thus encompasses the standard MDP planning as a special case. For $(\mathbf{s},\mathbf{a})$-rectangular uncertainty sets, we develop a policy-based first-order method, namely the robust policy mirror descent (RPMD), and establish an $\mathcal{O}(\log(1/\epsilon))$ and $\mathcal{O}(1/\epsilon)$ iteration complexity for finding an $\epsilon$-optimal policy, with two increasing-stepsize schemes. The prior convergence of RPMD is applicable to any Bregman divergence, provided the policy space has bounded radius measured by the divergence when centering at the initial policy. Moreover, when the Bregman divergence corresponds to the squared euclidean distance, we establish an $\mathcal{O}(\max \{1/\epsilon, 1/(\eta \epsilon^2)\})$ complexity of RPMD with any constant stepsize $\eta$. For a general class of Bregman divergences, a similar complexity is also established for RPMD with constant stepsizes, provided the uncertainty set satisfies the relative strong convexity. We further develop a stochastic variant, named SRPMD, when the first-order information is only available through online interactions with the nominal environment. For general Bregman divergences, we establish an $\mathcal{O}(1/\epsilon^2)$ and $\mathcal{O}(1/\epsilon^3)$ sample complexity with two increasing-stepsize schemes. For the euclidean Bregman divergence, we establish an $\mathcal{O}(1/\epsilon^3)$ sample complexity with constant stepsizes. To the best of our knowledge, all the aforementioned results appear to be new for policy-based first-order methods applied to the robust MDP problem.

Non-asymptotic Performances of Robust Markov Decision Processes

Toward Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptotics

Improving Robust Decisions with Data

Multistage Robust Mixed-Integer Optimization under Endogenous Uncertainty

Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets

Towards Minimax Optimality of Model-based Robust Reinforcement Learning

Robust Markov Decision Processes without Model Estimation

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

Robust Average-Reward Markov Decision Processes

Sample Complexity of Robust Reinforcement Learning with a Generative Model

Roping in Uncertainty: Robustness and Regularization in Markov Games

Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model

Avoiding Model Estimation in Robust Markov Decision Processes with a Generative Model

A Family of [math]-Rectangular Robust MDPs: Relative Conservativeness, Asymptotic Analyses, and Finite-Sample Properties

Efficient Policy Iteration for Robust Markov Decision Processes via Regularization

First-order Policy Optimization for Robust Markov Decision Process

Robust Batch Policy Learning in Markov Decision Processes

The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model

The Optimal Robust Control Policy for Uncertain Semi-Markov Control Processes

Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes