Abstract:We propose a partial differential-integral equation (PDE) framework for deep neural networks (DNNs) and their associated learning problem by taking the continuum limits of both network width and depth. The proposed model captures the complex interactions among hidden nodes, overcoming limitations of traditional discrete and ordinary differential equation (ODE)-based models. We explore the well-posedness of the forward propagation problem, analyze the existence and properties of minimizers for the learning task, and provide a detailed examination of necessary and sufficient conditions for the existence of critical points. Controllability and optimality conditions for the learning task with its associated PDE forward problem are established using variational calculus, the Pontryagin Maximum Principle, and the Hamilton-Jacobi-Bellman equation, framing the deep learning process as a PDE-constrained optimization problem. In this context, we prove the existence of viscosity solutions for the latter and we establish optimal feedback controls based on the value functional. This approach facilitates the development of new network architectures and numerical methods that improve upon traditional layer-by-layer gradient descent techniques by introducing forward-backward PDE discretization. The paper provides a mathematical foundation for connecting neural networks, PDE theory, variational analysis, and optimal control, partly building on and extending the results of \cite{liu2020selection}, where the main focus was the analysis of the forward evolution. By integrating these fields, we offer a robust framework that enhances deep learning models' stability, efficiency, and interpretability.

Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory

Two-scale Neural Networks for Partial Differential Equations with Small Parameters

A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

Some elliptic second order problems and neural network solutions: Existence and error estimates

Transferable Neural Networks for Partial Differential Equations

Optimally weighted loss functions for solving PDEs with Neural Networks

A Comparative Analysis of Optimization and Generalization Properties of Two-Layer Neural Network and Random Feature Models under Gradient Descent Dynamics

Constructing General Partial Differential Equations Using Polynomial and Neural Networks.

PDE Models for Deep Neural Networks: Learning Theory, Calculus of Variations and Optimal Control

A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations

A multi-scale framework for neural network enhanced methods to the solution of partial differential equations

A Second-Order Network Structure Based on Gradient-Enhanced Physics-Informed Neural Networks for Solving Parabolic Partial Differential Equations.

A Discussion on Solving Partial Differential Equations using Neural Networks

PDE-constrained Models with Neural Network Terms: Optimization and Global Convergence

A Least-Squares-Based Neural Network (LS-Net) for Solving Linear Parametric PDEs

A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

Theoretical properties of the global optimizer of two layer neural network

Learning Partial Differential Equations from Data Using Neural Networks

On the approximation of the solution of partial differential equations by artificial neural networks trained by a multilevel Levenberg-Marquardt method

TransNet: Transferable Neural Networks for Partial Differential Equations