Abstract:SIAM Journal on Scientific Computing, Volume 46, Issue 5, Page C535-C556, October 2024. We present a neural network approach for approximating the value function of high-dimensional stochastic control problems. Our training process simultaneously updates our value function estimate and identifies the part of the state space likely to be visited by optimal trajectories. Our approach leverages insights from optimal control theory and the fundamental relation between semilinear parabolic partial differential equations and forward-backward stochastic differential equations. To focus the sampling on relevant states during neural network training, we use the stochastic Pontryagin maximum principle (PMP) to obtain the optimal controls for the current value function estimate. By design, our approach coincides with the method of characteristics for the nonviscous Hamilton–Jacobi–Bellman equation arising in deterministic control problems. Our training loss consists of a weighted sum of the objective functional of the control problem and penalty terms that enforce the HJB equations along the sampled trajectories. Importantly, training is unsupervised in that it does not require solutions of the control problem. Our numerical experiments highlight our scheme's ability to identify the relevant parts of the state space and produce meaningful value estimates. Using a two-dimensional model problem, we demonstrate the importance of the stochastic PMP to inform the sampling and compare it to a finite element approach. With a nonlinear control affine quadcopter example, we illustrate that our approach can handle complicated dynamics. For a 100-dimensional benchmark problem, we demonstrate that our approach improves accuracy and time-to-solution, and, via a modification, we show the wider applicability of our scheme. Reproducibility of computational results.This paper has been awarded the "SIAM Reproducibility Badge: Code and data available" as recognition that the authors have followed reproducibility principles valued by SISC and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available at https://github.com/EmoryMLIP/NeuralSOC and in the supplementary material (NeuralSOC-main.zip [ 29.9MB]).

Representation Results and Error Estimates for Differential Games with Applications Using Neural Networks

Representation results and error estimates for differential games with applications using neural networks

Stochastic Delay Differential Games: Financial Modeling and Machine Learning Algorithms

A Neural Network Approach for Stochastic Optimal Control

Differential-game for resource aware approximate optimal control of large-scale nonlinear systems with multiple players

Numerical approximation of Dynkin games with asymmetric information

Neural networks-based algorithms for stochastic control and PDEs in finance

Linear Supervision for Nonlinear, High-Dimensional Neural Control and Differential Games

Penalty-Regulated Dynamics and Robust Learning Procedures in Games

Near-optimal control of dynamical systems with neural ordinary differential equations

Learning to Control Unknown Strongly Monotone Games

Deep Backward and Galerkin Methods for the Finite State Master Equation

Approximate-optimal control algorithm for constrained zero-sum differential games through event-triggering mechanism

Finite-horizon Equilibria for Neuro-symbolic Concurrent Stochastic Games

A unified stochastic approximation framework for learning in games

Deep Learning Approximation for Stochastic Control Problems.

A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

Newton’s Method, Bellman Recursion and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Differential Graphical Games for Constrained Autonomous Vehicles Based on Viability Theory

Semiglobal optimal feedback stabilization of autonomous systems via deep neural network approximation

Pareto Optimal Cooperative Control of Mean-Field Backward Stochastic Differential System in Finite Horizon