Abstract:Policy optimization (PO) is a key ingredient for reinforcement learning (RL). For control design, certain constraints are usually enforced on the policies to optimize, accounting for either the stability, robustness, or safety concerns on the system. Hence, PO is by nature a constrained (nonconvex) optimization in most cases, whose global convergence is challenging to analyze in general. More importantly, some constraints that are safety-critical, e.g., the $\mathcal{H}_\infty$-norm constraint that guarantees the system robustness, are difficult to enforce as the PO methods proceed. Recently, policy gradient methods have been shown to converge to the global optimum of linear quadratic regulator (LQR), a classical optimal control problem, without regularizing/projecting the control iterates onto the stabilizing set, its (implicit) feasible set. This striking result is built upon the coercive property of the cost, ensuring that the iterates remain feasible as the cost decreases. In this paper, we study the convergence theory of PO for $\mathcal{H}_2$ linear control with $\mathcal{H}_\infty$-norm robustness guarantee. One significant new feature of this problem is the lack of coercivity, i.e., the cost may have finite value around the feasible set boundary, breaking the existing analysis for LQR. Interestingly, we show that two PO methods enjoy the implicit regularization property, i.e., the iterates preserve the $\mathcal{H}_\infty$ robustness constraint as if they are regularized by the algorithms. Furthermore, despite the nonconvexity of the problem, we show that these algorithms converge to the globally optimal policies with globally sublinear rates, avoiding all suboptimal stationary points/local minima, and with locally (super-)linear rates under certain conditions.

Learning Stabilizing Controllers of Linear Systems via Discount Policy Gradient

Convergence and Sample Complexity of Policy Gradient Methods for Stabilizing Linear Systems

Output stabilization of boundary-controlled parabolic PDEs via gradient-based dynamic optimization

Stochastic stabilization of quasi-generalized Hamiltonian systems

Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators

Convergence of Policy Gradient for Stochastic Linear-Quadratic Control Problem in Infinite Horizon

Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradient

Stochastic Cubic-Regularized Policy Gradient Method

Logic-Based Switching Control for Stabilization of Stochastic Feedforward Nonlinear Systems

Revisiting LQR Control from the Perspective of Receding-Horizon Policy Gradient

Is Stabilizing an Unknown Linear System Easier Than Model Identification

Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies

Globally Convergent Policy Gradient Methods for Linear Quadratic Control of Partially Observed Systems

Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

Stochastic Reinforcement Learning with Stability Guarantees for Control of Unknown Nonlinear Systems

Policy Optimization for $\mathcal{H}_2$ Linear Control with $\mathcal{H}_\infty$ Robustness Guarantee: Implicit Regularization and Global Convergence

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergence

Optimization Landscape of Policy Gradient Methods for Discrete-Time Static Output Feedback

Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint

Asynchronous Heterogeneous Linear Quadratic Regulator Design