Abstract:We consider the infinite-horizon LQR control problem. Motivated by competitive analysis in online learning, as a criterion for controller design we introduce the dynamic regret, defined as the difference between the LQR cost of a causal controller (that has only access to past disturbances) and the LQR cost of the \emph{unique} clairvoyant one (that has also access to future disturbances) that is known to dominate all other controllers. The regret itself is a function of the disturbances, and we propose to find a causal controller that minimizes the worst-case regret over all bounded energy disturbances. The resulting controller has the interpretation of guaranteeing the smallest regret compared to the best non-causal controller that can see the future. We derive explicit formulas for the optimal regret and for the regret-optimal controller for the state-space setting. These explicit solutions are obtained by showing that the regret-optimal control problem can be reduced to a Nehari extension problem that can be solved explicitly. The regret-optimal controller is shown to be linear and can be expressed as the sum of the classical $H_2$ state-feedback law and an $n$-th order controller ($n$ is the state dimension), and its construction simply requires a solution to the standard LQR Riccati equation and two Lyapunov equations. Simulations over a range of plants demonstrate that the regret-optimal controller interpolates nicely between the $H_2$ and the $H_\infty$ optimal controllers, and generally has $H_2$ and $H_\infty$ costs that are simultaneously close to their optimal values. The regret-optimal controller thus presents itself as a viable option for control systems design.

LQR Control with Sparse Adversarial Disturbances

The Power of Linear Controllers in LQR Control

Distributionally Robust Linear Quadratic Control

Stochastic LQR Design With Disturbance Preview

The Distributionally Robust Infinite-Horizon LQR

Suboptimality analysis of receding horizon quadratic control with unknown linear systems and its applications in learning-based control

Online Control of Unknown Time-Varying Dynamical Systems

Asynchronous Heterogeneous Linear Quadratic Regulator Design

Linear Quadratic Control with Risk Constraints

Observation-based Optimal Control Law Learning with LQR Reconstruction

Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control

Generating Adversarial Disturbances for Controller Verification

Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator

Direct Data-Driven Discounted Infinite Horizon Linear Quadratic Regulator with Robustness Guarantees

Online Control with Adversarial Disturbance for Continuous-time Linear Systems

Regret-Optimal LQR Control

Handling Heterogeneous Curvatures in Bandit LQR Control

Exponentially Stable Adaptive Optimal Control of Uncertain LTI Systems

Distributionally Robust LQG control under Distributed Uncertainty

Stronger Regret Bounds for Safe Online Reinforcement Learning in the Linear Quadratic Regulator