Abstract:This article presents a reinforcement learning framework for continuous-time dynamical systems without a priori discretization of time, state, and action. Based on the Hamilton-Jacobi-Bellman (HJB) equation for infinite-horizon, discounted reward problems, we derive algorithms for estimating value functions and improving policies with the use of function approximators. The process of value function estimation is formulated as the minimization of a continuous-time form of the temporal difference (TD) error. Update methods based on backward Euler approximation and exponential eligibility traces are derived, and their correspondences with the conventional residual gradient, TD(0), and TD(lambda) algorithms are shown. For policy improvement, two methods-a continuous actor-critic method and a value-gradient-based greedy policy-are formulated. As a special case of the latter, a nonlinear feedback control law using the value gradient and the model of the input gain is derived. The advantage updating, a model-free algorithm derived previously, is also formulated in the HJB-based framework. The performance of the proposed algorithms is first tested in a nonlinear control task of swinging a pendulum up with limited torque. It is shown in the simulations that (1) the task is accomplished by the continuous actor-critic method in a number of trials several times fewer than by the conventional discrete actor-critic method; (2) among the continuous policy update methods, the value-gradient-based policy with a known or learned dynamic model performs several times better than the actor-critic method; and (3) a value function update using exponential eligibility traces is more efficient and stable than that based on Euler approximation. The algorithms are then tested in a higher-dimensional task: cart-pole swing-up. This task is accomplished in several hundred trials using the value-gradient-based policy with a learned dynamic model.

On the stability of Lipschitz continuous control problems and its application to reinforcement learning

Direct control method for improving stability and reliability of nonlinear stochastic dynamical systems

Optimal Control of Unbounded Functional Stochastic Evolution Systems in Hilbert Spaces: Second-Order Path-dependent HJB Equation

Stochastic optimal control in Hilbert spaces: $C^{1,1}$ regularity of the value function and optimal synthesis via viscosity solutions

A Model-free Framework for Nonlinear Optimal Control Based on Solving HJB Equation

Pathwise Relaxed Optimal Control of Rough Differential Equations

Lipschitz continuity and semiconcavity properties of the value function of a stochastic control problem

A unified framework of rapid exponential stability and optimal feedback control for nonlinear systems

Reinforcement learning in continuous time and space

Nonlinear Optimal Control for Stochastic Dynamical Systems

Stochastic Reinforcement Learning with Stability Guarantees for Control of Unknown Nonlinear Systems

Finite-Time Stabilization and Optimal Feedback Control for Nonlinear Discrete-Time Systems

Regularity and stability of feedback relaxed controls

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

Stability-certified reinforcement learning: A control-theoretic perspective

H_∞ Model-free Reinforcement Learning with Robust Stability Guarantee

Online adaptive learning of optimal control solutions using integral reinforcement learning

Robust Tracking Control with Reinforcement Learning for Nonlinear‐constrained Systems

Relaxation and asymptotic expansion of controlled stiff differential equations

Optimality conditions and Lipschitz stability for non-smooth semilinear elliptic optimal control problems with sparse controls