Abstract:The past decade has seen increasing interest in applying Deep Learning (DL) to Computational Science and Engineering (CSE). Driven by impressive results in applications such as computer vision, Uncertainty Quantification (UQ), genetics, simulations and image processing, DL is increasingly supplanting classical algorithms, and seems poised to revolutionize scientific computing. However, DL is not yet well-understood from the standpoint of numerical analysis. Little is known about the efficiency and reliability of DL from the perspectives of stability, robustness, accuracy, and sample complexity. In particular, approximating solutions to parametric PDEs is an objective of UQ for CSE. Training data for such problems is often scarce and corrupted by errors. Moreover, the target function is a possibly infinite-dimensional smooth function taking values in the PDE solution space, generally an infinite-dimensional Banach space. This paper provides arguments for Deep Neural Network (DNN) approximation of such functions, with both known and unknown parametric dependence, that overcome the curse of dimensionality. We establish practical existence theorems that describe classes of DNNs with dimension-independent architecture size and training procedures based on minimizing the (regularized) $\ell^2$-loss which achieve near-optimal algebraic rates of convergence. These results involve key extensions of compressed sensing for Banach-valued recovery and polynomial emulation with DNNs. When approximating solutions of parametric PDEs, our results account for all sources of error, i.e., sampling, optimization, approximation and physical discretization, and allow for training high-fidelity DNN approximations from coarse-grained sample data. Our theoretical results fall into the category of non-intrusive methods, providing a theoretical alternative to classical methods for high-dimensional approximation.

How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learning

Breaking the Curse of Dimensionality with Convex Neural Networks

Deep neural network approximation of composite functions without the curse of dimensionality

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff

Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks

Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm

Deep Neural Network Approximation of Composition Functions: with Application to PINNs

Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks

Compositional Sparsity, Approximation Classes, and Parametric Transport Equations

Learning Functions: When Is Deep Better Than Shallow

Spatially heterogeneous learning by a deep student machine

Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

The Kolmogorov Superposition Theorem can Break the Curse of Dimensionality When Approximating High Dimensional Functions

A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients

Lower bounds for artificial neural network approximations: A proof that shallow neural networks fail to overcome the curse of dimensionality

Revealing the Structure of Deep Neural Networks via Convex Duality

On the hardness of learning under symmetries

What can be learnt with wide convolutional neural networks? *