Abstract:SIAM Journal on Scientific Computing, Volume 46, Issue 5, Page C535-C556, October 2024. We present a neural network approach for approximating the value function of high-dimensional stochastic control problems. Our training process simultaneously updates our value function estimate and identifies the part of the state space likely to be visited by optimal trajectories. Our approach leverages insights from optimal control theory and the fundamental relation between semilinear parabolic partial differential equations and forward-backward stochastic differential equations. To focus the sampling on relevant states during neural network training, we use the stochastic Pontryagin maximum principle (PMP) to obtain the optimal controls for the current value function estimate. By design, our approach coincides with the method of characteristics for the nonviscous Hamilton–Jacobi–Bellman equation arising in deterministic control problems. Our training loss consists of a weighted sum of the objective functional of the control problem and penalty terms that enforce the HJB equations along the sampled trajectories. Importantly, training is unsupervised in that it does not require solutions of the control problem. Our numerical experiments highlight our scheme's ability to identify the relevant parts of the state space and produce meaningful value estimates. Using a two-dimensional model problem, we demonstrate the importance of the stochastic PMP to inform the sampling and compare it to a finite element approach. With a nonlinear control affine quadcopter example, we illustrate that our approach can handle complicated dynamics. For a 100-dimensional benchmark problem, we demonstrate that our approach improves accuracy and time-to-solution, and, via a modification, we show the wider applicability of our scheme. Reproducibility of computational results.This paper has been awarded the "SIAM Reproducibility Badge: Code and data available" as recognition that the authors have followed reproducibility principles valued by SISC and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available at https://github.com/EmoryMLIP/NeuralSOC and in the supplementary material (NeuralSOC-main.zip [ 29.9MB]).

Neural network-based finite horizon stochastic optimal control design for nonlinear networked control systems

Neural network-based finite-horizon optimal control of uncertain affine nonlinear discrete-time systems

Neural-Based Decentralized Adaptive Finite-Time Control for Nonlinear Large-Scale Systems With Time-Varying Output Constraints

A neural-network-based online optimal control approach for nonlinear robust decentralized stabilization

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Constrained Online Optimal Control for Continuous-Time Nonlinear Systems Using Neuro-Dynamic Programming

Neural-network-based Approach to Finite-Time Optimal Control for a Class of Unknown Nonlinear Systems

Finite-horizon Neural Optimal Tracking Control for a Class of Nonlinear Systems with Unknown Dynamics

Neural-network-based robust optimal control design for a class of uncertain nonlinear systems via adaptive dynamic programming

Quantized Neural Adaptive Finite-Time Preassigned Performance Control for Interconnected Nonlinear Systems

Fraction Dynamic-Surface-based Adaptive Neural Finite-Time Control for Stochastic Nonlinear Systems Subject to Unknown Control Directions, Time-Varying Input Delay and State Delay

Decentralized Adaptive Neural Inverse Optimal Control of Nonlinear Interconnected Systems

A Neural Network Approach for Stochastic Optimal Control

Online Adaptive Optimal Control for Continuous-Time Nonlinear Systems with Completely Unknown Dynamics.

A Fast Finite-Time Neural Network Control of Stochastic Nonlinear Systems

Adaptive Neural-Network-Based Control for a Class of Nonlinear Systems with Unknown Output Disturbance and Time Delays.

A novel adaptive neural predefined time control for a class of nonlinear systems

Adaptive Neural Networks Finite-Time Optimal Control for a Class of Nonlinear Systems

Neural Network-Based Adaptive Decentralized Learning Control for Interconnected Systems with Input Constraints

Adaptive NN control for a class of strict-feedback nonlinear systems with time delays