Abstract:SIAM Journal on Scientific Computing, Volume 46, Issue 5, Page C535-C556, October 2024. We present a neural network approach for approximating the value function of high-dimensional stochastic control problems. Our training process simultaneously updates our value function estimate and identifies the part of the state space likely to be visited by optimal trajectories. Our approach leverages insights from optimal control theory and the fundamental relation between semilinear parabolic partial differential equations and forward-backward stochastic differential equations. To focus the sampling on relevant states during neural network training, we use the stochastic Pontryagin maximum principle (PMP) to obtain the optimal controls for the current value function estimate. By design, our approach coincides with the method of characteristics for the nonviscous Hamilton–Jacobi–Bellman equation arising in deterministic control problems. Our training loss consists of a weighted sum of the objective functional of the control problem and penalty terms that enforce the HJB equations along the sampled trajectories. Importantly, training is unsupervised in that it does not require solutions of the control problem. Our numerical experiments highlight our scheme's ability to identify the relevant parts of the state space and produce meaningful value estimates. Using a two-dimensional model problem, we demonstrate the importance of the stochastic PMP to inform the sampling and compare it to a finite element approach. With a nonlinear control affine quadcopter example, we illustrate that our approach can handle complicated dynamics. For a 100-dimensional benchmark problem, we demonstrate that our approach improves accuracy and time-to-solution, and, via a modification, we show the wider applicability of our scheme. Reproducibility of computational results.This paper has been awarded the "SIAM Reproducibility Badge: Code and data available" as recognition that the authors have followed reproducibility principles valued by SISC and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available at https://github.com/EmoryMLIP/NeuralSOC and in the supplementary material (NeuralSOC-main.zip [ 29.9MB]).

Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Decentralized Safe Multi-agent Stochastic Optimal Control using Deep FBSDEs and ADMM

A Neural Network Approach for Stochastic Optimal Control

Safety of Dynamical Systems with Multiple Non-Convex Unsafe Sets Using Control Barrier Functions

Safety Guarantees for Neural Network Dynamic Systems via Stochastic Barrier Functions

Adaptive Deep Neural Network-Based Control Barrier Functions

Safe adaptive output‐feedback optimal control of a class of linear systems

Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions

Safe Non-Stochastic Control of Control-Affine Systems: An Online Convex Optimization Approach

Path Integral Methods with Stochastic Control Barrier Functions

Safe Neural Control for Non-Affine Control Systems with Differentiable Control Barrier Functions

State Constrained Stochastic Optimal Control for Continuous and Hybrid Dynamical Systems Using DFBSDE

Cooptimizing Safety and Performance with a Control-Constrained Formulation

Iterative Convex Optimization for Safety-Critical Model Predictive Control

Iterative Convex Optimization for Model Predictive Control with Discrete-Time High-Order Control Barrier Functions

Constrained Optimal Control for Nonlinear Multi-Input Safety-Critical Systems with Time-Varying Safety Constraints

Safe Stochastic Model Predictive Control

Online Optimization of Dynamical Systems with Deep Learning Perception

Bayesian Learning-Based Adaptive Control for Safety Critical Systems

Convergence of the deep BSDE method for stochastic control problems formulated through the stochastic maximum principle