Abstract:We propose a machine learning algorithm for solving finite-horizon stochastic control problems based on a deep neural network representation of the optimal policy functions. The algorithm has three features: (1) It can solve high-dimensional (e.g., over 100 dimensions) and finite-horizon time-inhomogeneous stochastic control problems. (2) It has a monotonicity of performance improvement in each iteration, leading to good convergence properties. (3) It does not rely on the Bellman equation. To demonstrate the efficiency of the algorithm, it is applied to solve various finite-horizon time-inhomogeneous problems including recursive utility optimization under a stochastic volatility model, a multi-sector stochastic growth, and optimal control under a dynamic stochastic integration of climate and economy model with eight-dimensional state vectors and 600 time periods.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to solve the stochastic control problem within a finite time - horizon in economics. Specifically, the author proposes a machine - learning algorithm based on the deep neural network to represent the optimal policy function, namely the Monotonic Monte - Carlo Control (MMCC) algorithm, in order to address the following challenges: 1. **High - dimensional and finite time - horizon**: Many stochastic control problems in economics are of a finite time - horizon and time - varying (time - inhomogeneous), which are more difficult to solve than the infinite time - horizon problems because the optimal control strategies in different time periods are different. 2. **The curse of dimensionality**: Due to high - dimensionality (for example, more than 100 dimensions) and complex stochastic dynamics, it is usually very difficult to numerically solve high - dimensional stochastic control problems. 3. **Limitations of the Bellman equation**: If the utility function in the control problem is not time - separable, then the problem may not have a Bellman equation, and thus the traditional dynamic programming method cannot be used. ### Solutions proposed in the paper To overcome the above difficulties, the author proposes the MMCC algorithm, whose main features include: - **Not relying on the Bellman equation**: The MMCC algorithm does not need to use the Bellman equation or the Euler equation, so it can handle a wider range of stochastic control problems. - **Monotonically improving performance**: In each iteration, the algorithm monotonically improves performance, thus having good convergence properties. - **Applicable to complex stochastic dynamics**: The algorithm allows the state evolution to have general stochastic dynamics, not just limited to discretized diffusion processes or Lévy processes. ### Application examples The paper demonstrates the effectiveness of the MMCC algorithm in multiple high - dimensional (such as more than 100 dimensions) stochastic control problems, including: - **Recursive utility optimization under the stochastic volatility model** - **Multi - sector stochastic growth problems** - **A stochastic integrated model of climate and economic dynamics**, where the state vector is 8 - dimensional and the time period is 600. Through these applications, the paper verifies the high efficiency and applicability of the MMCC algorithm in solving complex economic problems. ### Summary This paper aims to provide a new and efficient machine - learning algorithm to solve high - dimensional, finite - time - horizon, and time - inhomogeneous stochastic control problems in economics, especially those problems that are not suitable for being solved by traditional dynamic programming methods.

A Machine Learning Algorithm for Finite-Horizon Stochastic Control Problems in Economics

Neural networks-based algorithms for stochastic control and PDEs in finance

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

EM Algorithm and Stochastic Control in Economics

Deep Learning Approximation for Stochastic Control Problems.

A Simulation-Free Deep Learning Approach to Stochastic Optimal Control

A Neural Network Approach for Stochastic Optimal Control

A deep learning method for solving stochastic optimal control problems driven by fully-coupled FBSDEs

Learning-Based Neural Dynamic Surface Predictive Control for MMC

Recent Developments in Machine Learning Methods for Stochastic Control and Games

Neural-network-based finite horizon optimal control for partially unknown linear continuous-time systems

Neural network-based finite horizon stochastic optimal control design for nonlinear networked control systems

Neural network-based finite-horizon optimal control of uncertain affine nonlinear discrete-time systems

A Multilevel Approach for Stochastic Nonlinear Optimal Control

Adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems

Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

Machine Learning and Hamilton-Jacobi-Bellman Equation for Optimal Decumulation: a Comparison Study

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control

A hybrid deep learning method for finite-horizon mean-field game problems

Deep Learning for Population-Dependent Controls in Mean Field Control Problems with Common Noise

Deep multitask neural networks for solving some stochastic optimal control problems