Abstract:The past decade has seen increasing interest in applying Deep Learning (DL) to Computational Science and Engineering (CSE). Driven by impressive results in applications such as computer vision, Uncertainty Quantification (UQ), genetics, simulations and image processing, DL is increasingly supplanting classical algorithms, and seems poised to revolutionize scientific computing. However, DL is not yet well-understood from the standpoint of numerical analysis. Little is known about the efficiency and reliability of DL from the perspectives of stability, robustness, accuracy, and sample complexity. In particular, approximating solutions to parametric PDEs is an objective of UQ for CSE. Training data for such problems is often scarce and corrupted by errors. Moreover, the target function is a possibly infinite-dimensional smooth function taking values in the PDE solution space, generally an infinite-dimensional Banach space. This paper provides arguments for Deep Neural Network (DNN) approximation of such functions, with both known and unknown parametric dependence, that overcome the curse of dimensionality. We establish practical existence theorems that describe classes of DNNs with dimension-independent architecture size and training procedures based on minimizing the (regularized) $\ell^2$-loss which achieve near-optimal algebraic rates of convergence. These results involve key extensions of compressed sensing for Banach-valued recovery and polynomial emulation with DNNs. When approximating solutions of parametric PDEs, our results account for all sources of error, i.e., sampling, optimization, approximation and physical discretization, and allow for training high-fidelity DNN approximations from coarse-grained sample data. Our theoretical results fall into the category of non-intrusive methods, providing a theoretical alternative to classical methods for high-dimensional approximation.

Approximating smooth functions by deep neural networks with sigmoid activation function

Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function

Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Why Deep Neural Networks for Function Approximation?

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Approximation smooth and sparse functions by deep neural networks without saturation

Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constants

Approximation Bounds by Neural Networks in L Ω P [-4Pt]

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions

Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks

Deep neural network approximations for Monte Carlo algorithms

Sharp Bounds on the Approximation Rates, Metric Entropy, and $n$-widths of Shallow Neural Networks

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

The approximation capabilities of Durrmeyer-type neural network operators

Deep Neural Network Approximation of Composition Functions: with Application to PINNs

An approximation by neural networkswith a fixed weight

Advantage of Deep Neural Networks for Estimating Functions with Singularity on Hypersurfaces