Abstract:The past decade has seen increasing interest in applying Deep Learning (DL) to Computational Science and Engineering (CSE). Driven by impressive results in applications such as computer vision, Uncertainty Quantification (UQ), genetics, simulations and image processing, DL is increasingly supplanting classical algorithms, and seems poised to revolutionize scientific computing. However, DL is not yet well-understood from the standpoint of numerical analysis. Little is known about the efficiency and reliability of DL from the perspectives of stability, robustness, accuracy, and sample complexity. In particular, approximating solutions to parametric PDEs is an objective of UQ for CSE. Training data for such problems is often scarce and corrupted by errors. Moreover, the target function is a possibly infinite-dimensional smooth function taking values in the PDE solution space, generally an infinite-dimensional Banach space. This paper provides arguments for Deep Neural Network (DNN) approximation of such functions, with both known and unknown parametric dependence, that overcome the curse of dimensionality. We establish practical existence theorems that describe classes of DNNs with dimension-independent architecture size and training procedures based on minimizing the (regularized) $\ell^2$-loss which achieve near-optimal algebraic rates of convergence. These results involve key extensions of compressed sensing for Banach-valued recovery and polynomial emulation with DNNs. When approximating solutions of parametric PDEs, our results account for all sources of error, i.e., sampling, optimization, approximation and physical discretization, and allow for training high-fidelity DNN approximations from coarse-grained sample data. Our theoretical results fall into the category of non-intrusive methods, providing a theoretical alternative to classical methods for high-dimensional approximation.

A duality framework for analyzing random feature and two-layer neural networks

A Duality Framework for Generalization Analysis of Random Feature Models and Two-Layer Neural Networks.

Strong Duality Relations in Nonconvex Risk-Constrained Learning

A Double-Variational Bayesian Framework in Random Fourier Features for Indefinite Kernels

Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks

Optimal Convergence for Agnostic Kernel Learning with Random Features.

Universal approximation property of Banach space-valued random feature models including random neural networks

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

A Duality Analysis of Kernel Ridge Regression in the Noiseless Regime

A Duality Approach to Regularized Learning Problems in Banach Spaces

Deep Kernel Learning Networks with Multiple Learning Paths

Embedding Inequalities for Barron-type Spaces

Duality of Bures and Shape Distances with Implications for Comparing Neural Representations

High-Dimensional Analysis for Generalized Nonlinear Regression: from Asymptotics to Algorithm

Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks

Machine learning and optimization-based approaches to duality in statistical physics

The generalization error of random features regression: Precise asymptotics and double descent curve

Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space

Random Features Approximation for Control-Affine Systems

On Duality Between Filtering And Interpolation

Information-Theoretic Foundations for Machine Learning