Abstract:SIAM Journal on Scientific Computing, Volume 46, Issue 5, Page C535-C556, October 2024. We present a neural network approach for approximating the value function of high-dimensional stochastic control problems. Our training process simultaneously updates our value function estimate and identifies the part of the state space likely to be visited by optimal trajectories. Our approach leverages insights from optimal control theory and the fundamental relation between semilinear parabolic partial differential equations and forward-backward stochastic differential equations. To focus the sampling on relevant states during neural network training, we use the stochastic Pontryagin maximum principle (PMP) to obtain the optimal controls for the current value function estimate. By design, our approach coincides with the method of characteristics for the nonviscous Hamilton–Jacobi–Bellman equation arising in deterministic control problems. Our training loss consists of a weighted sum of the objective functional of the control problem and penalty terms that enforce the HJB equations along the sampled trajectories. Importantly, training is unsupervised in that it does not require solutions of the control problem. Our numerical experiments highlight our scheme's ability to identify the relevant parts of the state space and produce meaningful value estimates. Using a two-dimensional model problem, we demonstrate the importance of the stochastic PMP to inform the sampling and compare it to a finite element approach. With a nonlinear control affine quadcopter example, we illustrate that our approach can handle complicated dynamics. For a 100-dimensional benchmark problem, we demonstrate that our approach improves accuracy and time-to-solution, and, via a modification, we show the wider applicability of our scheme. Reproducibility of computational results.This paper has been awarded the "SIAM Reproducibility Badge: Code and data available" as recognition that the authors have followed reproducibility principles valued by SISC and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available at https://github.com/EmoryMLIP/NeuralSOC and in the supplementary material (NeuralSOC-main.zip [ 29.9MB]).

Semiglobal optimal feedback stabilization of autonomous systems via deep neural network approximation

Neural Network Optimal Feedback Control with Guaranteed Local Stability

On the stability analysis of deep neural network representations of an optimal state-feedback

Adaptive Neural Network Control for Nonlinear Systems Based on Approximation Errors

Deep neural network approximations for the stable manifolds of the Hamilton-Jacobi-Bellman equations

A neural-network-based online optimal control approach for nonlinear robust decentralized stabilization

Neural network optimal feedback control with enhanced closed loop stability

Globally Stable Adaptive Tracking Control for Uncertain Strict-Feedback Systems Based on Neural Network Approximation

A Neural Network Approach for Stochastic Optimal Control

Decentralized Stabilization for a Class of Continuous-Time Nonlinear Interconnected Systems Using Online Learning Optimal Control Approach

Robust Adaptive Neural Network Control for a Class of Nonlinear Systems

An approach to neural control of a class of strict-feedback nonlinear systems

Adaptive Neural Network Optimal Backstepping Control of Strict Feedback Nonlinear Systems Via Reinforcement Learning

Stable Tracking Control to a Nonlinear Process Via Neural Network Model

Approximation-Based Adaptive Neural Control Design for A Class of Nonlinear Systems

Observer-based Nonlinear Feedback Decentralized Neural Adaptive Dynamic Surface Control for Large-Scale Nonlinear Systems

Adaptive Neural Control for a Class of Perturbed Strict-Feedback Nonlinear Time-Delay Systems

Robust adaptive neural control for strict-feedback nonlinear systems

Neural Control Systems

Optimal distributed control with stability guarantees by training a network of neural closed-loop maps

Decentralized Adaptive Neural Inverse Optimal Control of Nonlinear Interconnected Systems