Abstract:SIAM Journal on Control and Optimization, Volume 62, Issue 1, Page 135-166, February 2024. This work uses the entropy-regularized relaxed stochastic control perspective as a principled framework for designing reinforcement learning (RL) algorithms. Herein, an agent interacts with the environment by generating noisy controls distributed according to the optimal relaxed policy. The noisy policies, on the one hand, explore the space and hence facilitate learning, but, on the other hand, they introduce bias by assigning a positive probability to nonoptimal actions. This exploration-exploitation trade-off is determined by the strength of entropy regularization. We study algorithms resulting from two entropy regularization formulations: the exploratory control approach, where entropy is added to the cost objective, and the proximal policy update approach, where entropy penalizes policy divergence between consecutive episodes. We focus on the finite horizon continuous-time linear-quadratic (LQ) RL problem, where a linear dynamics with unknown drift coefficients is controlled subject to quadratic costs. In this setting, both algorithms yield a Gaussian relaxed policy. We quantify the precise difference between the value functions of a Gaussian policy and its noisy evaluation and show that the execution noise must be independent across time. By tuning the frequency of sampling from relaxed policies and the parameter governing the strength of entropy regularization, we prove that the regret, for both learning algorithms, is of the order [math] (up to a logarithmic factor) over [math] episodes, matching the best known result from the literature.

Reinforcement Learning for a Discrete-Time Linear-Quadratic Control Problem with an Application

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Sublinear Regret for a Class of Continuous-Time Linear--Quadratic Reinforcement Learning Problems

Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning

Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator

Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control

A Reinforcement Learning Method for LQR Control Problem

Robust policy iteration for continuous-time stochastic $H_\infty$ control problem with unknown dynamics

A Tour of Reinforcement Learning: The View from Continuous Control

Adaptive Optimal Control of Discrete-Time Linear Systems with Discounted Value: Off-Policy Reinforcement Learning

H∞ Tracking Control for Linear Discrete‐time Systems Via Reinforcement Learning

Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning

Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty

Reinforcement Learning for Jump-Diffusions, with Financial Applications

Continuous‐time mean–variance portfolio selection: A reinforcement learning framework

Robust Control of Uncertain Linear Systems Based on Reinforcement Learning Principles.

Reinforcement Learning for Finite-Horizon H∞ Tracking Control of Unknown Discrete Linear Time-Varying System

Data-Driven Robust Control of Discrete-Time Uncertain Linear Systems Via Off-Policy Reinforcement Learning.