Abstract:Numerically solving high-dimensional partial differential equations (PDEs) is a major challenge. Conventional methods, such as finite difference methods, are unable to solve high-dimensional PDEs due to the curse-of-dimensionality. A variety of deep learning methods have been recently developed to try and solve high-dimensional PDEs by approximating the solution using a neural network. In this paper, we prove global convergence for one of the commonly-used deep learning algorithms for solving PDEs, the Deep Galerkin Method (DGM). DGM trains a neural network approximator to solve the PDE using stochastic gradient descent. We prove that, as the number of hidden units in the single-layer network goes to infinity (i.e., in the ``wide network limit"), the trained neural network converges to the solution of an infinite-dimensional linear ordinary differential equation (ODE). The PDE residual of the limiting approximator converges to zero as the training time $\rightarrow \infty$. Under mild assumptions, this convergence also implies that the neural network approximator converges to the solution of the PDE. A closely related class of deep learning methods for PDEs is Physics Informed Neural Networks (PINNs). Using the same mathematical techniques, we can prove a similar global convergence result for the PINN neural network approximators. Both proofs require analyzing a kernel function in the limit ODE governing the evolution of the limit neural network approximator. A key technical challenge is that the kernel function, which is a composition of the PDE operator and the neural tangent kernel (NTK) operator, lacks a spectral gap, therefore requiring a careful analysis of its properties.

Convergence Rate Analysis for Deep Ritz Method

Error Analysis of Deep Ritz Methods for Elliptic Equations.

Deep Ritz Methods for Laplace Equations with Dirichlet Boundary Condition

Convergence Analysis of a Quasi-Monte Carlo-based Deep Learning Algorithm for Solving Partial Differential Equations

Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method

Statistical Numerical PDE : Fast Rate, Neural Scaling Law and When it’s Optimal

Convergence Analysis for the PINNs

Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality

DRM Revisited: A Complete Error Analysis

Convergence Analysis of the Deep Galerkin Method for Weak Solutions

A Priori Error Estimate of Deep Mixed Residual Method for Elliptic PDEs

Analysis of Deep Ritz Methods for Semilinear Elliptic Equations

A Rate of Convergence of Physics Informed Neural Networks for the Linear Second Order Elliptic PDEs

Refined generalization analysis of the Deep Ritz Method and Physics-Informed Neural Networks

Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equations

A Deep Double Ritz Method (D$^2$RM) for solving Partial Differential Equations using Neural Networks

Deep Ritz Method with Adaptive Quadrature for Linear Elasticity

A comparison study of deep Galerkin method and deep Ritz method for elliptic problems with different boundary conditions

Convergence Analysis of Deep Residual Networks

Convergence Analysis for Over-Parameterized Deep Learning