Abstract:Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the $\mathcal{H}_\infty$-norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, policy gradient methods have been shown to converge to the global optimum of linear quadratic regulator (LQR), a classical optimal control problem, without regularizing/projecting the control iterates onto the stabilizing set, its (implicit) feasible set. This striking result is built upon the coercive property of the cost, ensuring that the iterates remain feasible as the cost decreases. In this paper, we study the convergence theory of PO for $\mathcal{H}_2$ linear control with $\mathcal{H}_\infty$-norm robustness guarantee. One significant new feature of this problem is the lack of coercivity, i.e., the cost may have finite value around the feasible set boundary, breaking the existing analysis for LQR. Interestingly, we show that two PO methods enjoy the implicit regularization property, i.e., the iterates preserve the $\mathcal{H}_\infty$ robustness constraint as if they are regularized by the algorithms. Furthermore, despite the nonconvexity of the problem, we show that these algorithms converge to the globally optimal policies with globally sublinear rates, avoiding all suboptimal stationary points/local minima, and with locally (super-)linear rates under certain conditions.

Policy Optimization over Submanifolds for Linearly Constrained Feedback Synthesis

Iterative design of suboptimal feedback control for bilinear parabolic PDE systems

Policy Optimization in Control: Geometry and Algorithmic Implications

Regret Analysis of Policy Optimization over Submanifolds for Linearly Constrained Online LQG

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Policy Optimization for $\mathcal{H}_2$ Linear Control with $\mathcal{H}_\infty$ Robustness Guarantee: Implicit Regularization and Global Convergence

Global Convergence of Direct Policy Search for State-Feedback $\mathcal{H}_\infty$ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential

Synthesis of constrained robust feedback policies and model predictive control

Output-feedback Synthesis Orbit Geometry: Quotient Manifolds and LQG Direct Policy Optimization

Optimality of Affine Policies in Multi-stage Robust Optimization

Symplectic Stiefel manifold: tractable metrics, second-order geometry and Newton's methods

Systematic Stabilization of Constrained Piecewise Affine Systems

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

Optimal Control of Constrained Stochastic Linear-Quadratic Model with Applications

Simultaneous Controller and Lyapunov Function Design for Constrained Nonlinear Systems

Multiobjective Control Synthesis Based on Parametric Eigenstructure Assignment

Optimal Control of Robotic Systems and Biased Riemannian Splines

Optimality conditions for the nonlinear programming problems on Riemannian manifolds

Benign Nonconvex Landscapes in Optimal and Robust Control, Part I: Global Optimality

Optimization on manifolds: A symplectic approach

Globally Convergent Policy Gradient Methods for Linear Quadratic Control of Partially Observed Systems