Abstract:The past decade has seen increasing interest in applying Deep Learning (DL) to Computational Science and Engineering (CSE). Driven by impressive results in applications such as computer vision, Uncertainty Quantification (UQ), genetics, simulations and image processing, DL is increasingly supplanting classical algorithms, and seems poised to revolutionize scientific computing. However, DL is not yet well-understood from the standpoint of numerical analysis. Little is known about the efficiency and reliability of DL from the perspectives of stability, robustness, accuracy, and sample complexity. In particular, approximating solutions to parametric PDEs is an objective of UQ for CSE. Training data for such problems is often scarce and corrupted by errors. Moreover, the target function is a possibly infinite-dimensional smooth function taking values in the PDE solution space, generally an infinite-dimensional Banach space. This paper provides arguments for Deep Neural Network (DNN) approximation of such functions, with both known and unknown parametric dependence, that overcome the curse of dimensionality. We establish practical existence theorems that describe classes of DNNs with dimension-independent architecture size and training procedures based on minimizing the (regularized) $\ell^2$-loss which achieve near-optimal algebraic rates of convergence. These results involve key extensions of compressed sensing for Banach-valued recovery and polynomial emulation with DNNs. When approximating solutions of parametric PDEs, our results account for all sources of error, i.e., sampling, optimization, approximation and physical discretization, and allow for training high-fidelity DNN approximations from coarse-grained sample data. Our theoretical results fall into the category of non-intrusive methods, providing a theoretical alternative to classical methods for high-dimensional approximation.

Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations

Stochastic Neural Networks with Infinite Width are Deterministic

Bayesian Neural Ordinary Differential Equations

Partially Stochastic Infinitely Deep Bayesian Neural Networks

SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates

Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification

Neural Ordinary Differential Equations with Envolutionary Weights

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

Graph Neural Stochastic Differential Equations

Inferring the Langevin Equation with Uncertainty via Bayesian Neural Networks

Efficient Inference on a Network of Spiking Neurons using Deep Learning

Bayesian Deep Learning Framework for Uncertainty Quantification in Stochastic Partial Differential Equations

Scalable Bayesian Inference in the Era of Deep Learning: From Gaussian Processes to Deep Neural Networks

Density regression and uncertainty quantification with Bayesian deep noise neural networks

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

Neural SDEs as Infinite-Dimensional GANs

Functional SDE approximation inspired by a deep operator network architecture

Fully Bayesian Differential Gaussian Processes through Stochastic Differential Equations

Learning effective stochastic differential equations from microscopic simulations: Linking stochastic numerics to deep learning.

Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations

A backward SDE method for uncertainty quantification in deep learning