Abstract:The past decade has seen increasing interest in applying Deep Learning (DL) to Computational Science and Engineering (CSE). Driven by impressive results in applications such as computer vision, Uncertainty Quantification (UQ), genetics, simulations and image processing, DL is increasingly supplanting classical algorithms, and seems poised to revolutionize scientific computing. However, DL is not yet well-understood from the standpoint of numerical analysis. Little is known about the efficiency and reliability of DL from the perspectives of stability, robustness, accuracy, and sample complexity. In particular, approximating solutions to parametric PDEs is an objective of UQ for CSE. Training data for such problems is often scarce and corrupted by errors. Moreover, the target function is a possibly infinite-dimensional smooth function taking values in the PDE solution space, generally an infinite-dimensional Banach space. This paper provides arguments for Deep Neural Network (DNN) approximation of such functions, with both known and unknown parametric dependence, that overcome the curse of dimensionality. We establish practical existence theorems that describe classes of DNNs with dimension-independent architecture size and training procedures based on minimizing the (regularized) $\ell^2$-loss which achieve near-optimal algebraic rates of convergence. These results involve key extensions of compressed sensing for Banach-valued recovery and polynomial emulation with DNNs. When approximating solutions of parametric PDEs, our results account for all sources of error, i.e., sampling, optimization, approximation and physical discretization, and allow for training high-fidelity DNN approximations from coarse-grained sample data. Our theoretical results fall into the category of non-intrusive methods, providing a theoretical alternative to classical methods for high-dimensional approximation.

Efficient Bayesian Updates for Deep Learning via Laplace Approximations

Accelerated Linearized Laplace Approximation for Bayesian Deep Learning

Variational Linearized Laplace Approximation for Bayesian Deep Learning

Efficient Variational Bayesian Model Updating by Bayesian Active Learning

FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning

Generalized Laplace Approximation

Scalable Bayesian Inference in the Era of Deep Learning: From Gaussian Processes to Deep Neural Networks

Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations

Efficient Weight-Space Laplace-Gaussian Filtering and Smoothing for Sequential Deep Learning

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

Bayesian Numerical Integration with Neural Networks

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

Fast Laplace Approximation for Sparse Bayesian Spike and Slab Models

Improving Neural Additive Models with Bayesian Principles

The Case for Bayesian Deep Learning

An adaptive Hessian approximated stochastic gradient MCMC method

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

Scalable Bayesian Learning with posteriors

Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models