Abstract:The past decade has seen increasing interest in applying Deep Learning (DL) to Computational Science and Engineering (CSE). Driven by impressive results in applications such as computer vision, Uncertainty Quantification (UQ), genetics, simulations and image processing, DL is increasingly supplanting classical algorithms, and seems poised to revolutionize scientific computing. However, DL is not yet well-understood from the standpoint of numerical analysis. Little is known about the efficiency and reliability of DL from the perspectives of stability, robustness, accuracy, and sample complexity. In particular, approximating solutions to parametric PDEs is an objective of UQ for CSE. Training data for such problems is often scarce and corrupted by errors. Moreover, the target function is a possibly infinite-dimensional smooth function taking values in the PDE solution space, generally an infinite-dimensional Banach space. This paper provides arguments for Deep Neural Network (DNN) approximation of such functions, with both known and unknown parametric dependence, that overcome the curse of dimensionality. We establish practical existence theorems that describe classes of DNNs with dimension-independent architecture size and training procedures based on minimizing the (regularized) $\ell^2$-loss which achieve near-optimal algebraic rates of convergence. These results involve key extensions of compressed sensing for Banach-valued recovery and polynomial emulation with DNNs. When approximating solutions of parametric PDEs, our results account for all sources of error, i.e., sampling, optimization, approximation and physical discretization, and allow for training high-fidelity DNN approximations from coarse-grained sample data. Our theoretical results fall into the category of non-intrusive methods, providing a theoretical alternative to classical methods for high-dimensional approximation.

Deep Radial Kernel Networks: Approximating Radially Symmetric Functions with Deep Networks

A Novel Radial Basis Function Neural Network For Approximation

Approximation by non-symmetric networks for cross-domain learning

Learning in Deep Radial Basis Function Networks

Interpolation and Best Approximation for Spherical Radial Basis Function Networks

Approximation of Function by Adaptively Growing Radial Basis Function Neural Networks.

Investigation of Neural Networks for Function Approximation

Numerical Linear Approximation Involving Radial Basis Functions

A Novel Rbf Neural Network With Fast Training And Accurate Generalization

Learning Explicit Deep Representations from Deep Kernel Networks

On Radial Basis Function Net and Kernel Regression: Approximation Ability, Convergence Rate and Receptive Field Size

Efficient kernel surrogates for neural network-based regression

Approximation of RKHS Functionals by Neural Networks

Universality and Optimality of Structured Deep Kernel Networks

A Deep Neural Network Based Quasi-Linear Kernel for Support Vector Machines.

Why Deep Neural Networks for Function Approximation?

Universal Approximation Using Radial-Basis-Function Networks

SKFAC: Training Neural Networks with Faster Kronecker-Factored Approximate Curvature

On the Approximation of Kernel functions

From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks