Abstract:Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the $\mathcal{H}_\infty$-norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, policy gradient methods have been shown to converge to the global optimum of linear quadratic regulator (LQR), a classical optimal control problem, without regularizing/projecting the control iterates onto the stabilizing set, its (implicit) feasible set. This striking result is built upon the coercive property of the cost, ensuring that the iterates remain feasible as the cost decreases. In this paper, we study the convergence theory of PO for $\mathcal{H}_2$ linear control with $\mathcal{H}_\infty$-norm robustness guarantee. One significant new feature of this problem is the lack of coercivity, i.e., the cost may have finite value around the feasible set boundary, breaking the existing analysis for LQR. Interestingly, we show that two PO methods enjoy the implicit regularization property, i.e., the iterates preserve the $\mathcal{H}_\infty$ robustness constraint as if they are regularized by the algorithms. Furthermore, despite the nonconvexity of the problem, we show that these algorithms converge to the globally optimal policies with globally sublinear rates, avoiding all suboptimal stationary points/local minima, and with locally (super-)linear rates under certain conditions.

Robust Policy Optimization in Continuous-time Mixed $\mathcal{H}_2/\mathcal{H}_\infty$ Stochastic Control

Policy Optimization for $\mathcal{H}_2$ Linear Control with $\mathcal{H}_\infty$ Robustness Guarantee: Implicit Regularization and Global Convergence

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Robust policy iteration for continuous-time stochastic $H_\infty$ control problem with unknown dynamics

Policy Optimization for Continuous Reinforcement Learning

Continuous-Time Policy Optimization.

Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control

Robust affine control of linear stochastic systems

On the Global Optimality of Direct Policy Search for Nonsmooth $H_\infty$ Output-Feedback Control

Benign Nonconvex Landscapes in Optimal and Robust Control, Part I: Global Optimality

Robust Reinforcement Learning for Continuous Control with Model Misspecification

Reinforcement Learning Policies in Continuous-Time Linear Systems

H_∞ Model-free Reinforcement Learning with Robust Stability Guarantee

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

Policy Iteration Algorithm for Online Design of Robust Control for a Class of Continuous-Time Nonlinear Systems

$$H_\infty $$ Control Using Reinforcement Learning

Policy Optimization for PDE Control with a Warm Start

Robust H∞ Control of Unknown Discrete-Time Linear Systems with Time-Varying Uncertainties

Off-Policy Risk-Sensitive Reinforcement Learning-Based Constrained Robust Optimal Control

A Novel Policy Iteration Algorithm for Nonlinear Continuous-Time H$\infty$ Control Problem

Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization