Abstract:SIAM Journal on Control and Optimization, Volume 62, Issue 1, Page 135-166, February 2024. This work uses the entropy-regularized relaxed stochastic control perspective as a principled framework for designing reinforcement learning (RL) algorithms. Herein, an agent interacts with the environment by generating noisy controls distributed according to the optimal relaxed policy. The noisy policies, on the one hand, explore the space and hence facilitate learning, but, on the other hand, they introduce bias by assigning a positive probability to nonoptimal actions. This exploration-exploitation trade-off is determined by the strength of entropy regularization. We study algorithms resulting from two entropy regularization formulations: the exploratory control approach, where entropy is added to the cost objective, and the proximal policy update approach, where entropy penalizes policy divergence between consecutive episodes. We focus on the finite horizon continuous-time linear-quadratic (LQ) RL problem, where a linear dynamics with unknown drift coefficients is controlled subject to quadratic costs. In this setting, both algorithms yield a Gaussian relaxed policy. We quantify the precise difference between the value functions of a Gaussian policy and its noisy evaluation and show that the execution noise must be independent across time. By tuning the frequency of sampling from relaxed policies and the parameter governing the strength of entropy regularization, we prove that the regret, for both learning algorithms, is of the order [math] (up to a logarithmic factor) over [math] episodes, matching the best known result from the literature.

Backward Stochastic Control System with Entropy Regularization

Direct control method for improving stability and reliability of nonlinear stochastic dynamical systems

Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning

An entropy penalized approach for stochastic control problems. Complete version

Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning

Regularity and stability of feedback relaxed controls

Optimal Variational Principle for Backward Stochastic Control Systems Associated with Lévy Processes

Exploratory Optimal Stopping: A Singular Control Formulation

Maximum Entropy Density Control of Discrete-Time Linear Systems with Quadratic Cost

Maximum principle for optimal control of stochastic evolution equations with recursive utilities

Partial Information Near-Optimal Control of Forward-Backward Stochastic Differential System with Observation Noise

Entropy-Regularized Stochastic Games

Tsallis Entropy Regularization for Linearly Solvable MDP and Linear Quadratic Regulator

Non-homogeneous stochastic LQ control with regime switching and random coefficients

Second Order Optimality Conditions for Optimal Control Problems of Stochastic Evolution Equations

Stochastic Optimization Theory of Backward Stochastic Differential Equations Driven by G-Brownian Motion

Infinite Horizon Backward Stochastic Difference Equations and Related Stochastic Recursive Control Problems

Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems

The Global Maximum Principle for Progressive Optimal Control of Partially Observed Forward-Backward Stochastic Systems with Random Jumps

A stochastic maximum principle for forward-backward stochastic control systems with quadratic generators and sample-wise constraints

Stochastic linear quadratic optimal control problems with regime-switching jumps in infinite horizon