Abstract:We propose a partial differential-integral equation (PDE) framework for deep neural networks (DNNs) and their associated learning problem by taking the continuum limits of both network width and depth. The proposed model captures the complex interactions among hidden nodes, overcoming limitations of traditional discrete and ordinary differential equation (ODE)-based models. We explore the well-posedness of the forward propagation problem, analyze the existence and properties of minimizers for the learning task, and provide a detailed examination of necessary and sufficient conditions for the existence of critical points. Controllability and optimality conditions for the learning task with its associated PDE forward problem are established using variational calculus, the Pontryagin Maximum Principle, and the Hamilton-Jacobi-Bellman equation, framing the deep learning process as a PDE-constrained optimization problem. In this context, we prove the existence of viscosity solutions for the latter and we establish optimal feedback controls based on the value functional. This approach facilitates the development of new network architectures and numerical methods that improve upon traditional layer-by-layer gradient descent techniques by introducing forward-backward PDE discretization. The paper provides a mathematical foundation for connecting neural networks, PDE theory, variational analysis, and optimal control, partly building on and extending the results of \cite{liu2020selection}, where the main focus was the analysis of the forward evolution. By integrating these fields, we offer a robust framework that enhances deep learning models' stability, efficiency, and interpretability.

Dynamical systems and optimal control approach to deep learning

Dynamical Systems and Optimal Control Approach to Deep Learning

A Mean-Field Optimal Control Formulation of Deep Learning

Maximum Principle Based Algorithms for Deep Learning.

Nonlinear Stochastic Dynamics and Control in Hamiltonian Formulation

Accelerated Optimization in Deep Learning with a Proportional-Integral-derivative Controller

Relationships Between the Maximum Principle and Dynamic Programming for Infinite Dimensional Stochastic Control Systems

Near-optimal control of dynamical systems with neural ordinary differential equations

PDE Models for Deep Neural Networks: Learning Theory, Calculus of Variations and Optimal Control

Nonlinear Optimal Control Based on FBDEs and its Application to AGV

Physics-informed neural networks via stochastic Hamiltonian dynamics learning

Recent Developments in Machine Learning Methods for Stochastic Control and Games

Mean-field Langevin System, Optimal Control and Deep Neural Networks

Symmetric Discrete Optimal Control and Deep Learning

Online Optimization of Dynamical Systems with Deep Learning Perception

Deep Learning Approximation for Stochastic Control Problems.

Deep Learning via Dynamical Systems: An Approximation Perspective

Physical Deep Learning Based on Optimal Control of Dynamical Systems

Controllability-Constrained Deep Network Models for Enhanced Control of Dynamical Systems

Optimal Control of Dynamical Systems using Calculus of Variations

Neural optimal controller for stochastic systems via pathwise HJB operator