Abstract:Direct policy search has been widely applied in modern reinforcement learning and continuous control. However, the theoretical properties of direct policy search on nonsmooth robust control synthesis have not been fully understood. The optimal $\mathcal{H}_\infty$ control framework aims at designing a policy to minimize the closed-loop $\mathcal{H}_\infty$ norm, and is arguably the most fundamental robust control paradigm. In this work, we show that direct policy search is guaranteed to find the global solution of the robust $\mathcal{H}_\infty$ state-feedback control design problem. Notice that policy search for optimal $\mathcal{H}_\infty$ control leads to a constrained nonconvex nonsmooth optimization problem, where the nonconvex feasible set consists of all the policies stabilizing the closed-loop dynamics. We show that for this nonsmooth optimization problem, all Clarke stationary points are global minimum. Next, we identify the coerciveness of the closed-loop $\mathcal{H}_\infty$ objective function, and prove that all the sublevel sets of the resultant policy search problem are compact. Based on these properties, we show that Goldstein's subgradient method and its implementable variants can be guaranteed to stay in the nonconvex feasible set and eventually find the global optimal solution of the $\mathcal{H}_\infty$ state-feedback synthesis problem. Our work builds a new connection between nonconvex nonsmooth optimization theory and robust control, leading to an interesting global convergence result for direct policy search on optimal $\mathcal{H}_\infty$ synthesis.

Convergence Guarantees for Differentiable Optimization-based Control Policy

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Globally Convergent Policy Gradient Methods for Linear Quadratic Control of Partially Observed Systems

On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control

Optimization Landscape of Policy Gradient Methods for Discrete-Time Static Output Feedback

A Combined Policy Gradient and Q-learning Method for Data-driven Optimal Control Problems

Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs

A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee

Continuous-Time Policy Optimization.

Safe Neural Control for Non-Affine Control Systems with Differentiable Control Barrier Functions

Global Convergence of Direct Policy Search for State-Feedback $\mathcal{H}_\infty$ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential

On the Global Optimality of Direct Policy Search for Nonsmooth $H_\infty$ Output-Feedback Control

Policy Optimization Adaptive Dynamic Programming for Optimal Control of Input-Affine Discrete-Time Nonlinear Systems.

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces

Learning Optimal Control Policy for Unknown Discrete-Time Systems

Optimal Learning Output Tracking Control: A Model-Free Policy Optimization Method With Convergence Analysis

Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators