Abstract:Neural operator architectures approximate operators between infinite-dimensional Banach spaces of functions. They are gaining increased attention in computational science and engineering, due to their potential both to accelerate traditional numerical methods and to enable data-driven discovery. As the field is in its infancy basic questions about minimal requirements for universal approximation remain open. It is clear that any general approximation of operators between spaces of functions must be both nonlocal and nonlinear. In this paper we describe how these two attributes may be combined in a simple way to deduce universal approximation. In so doing we unify the analysis of a wide range of neural operator architectures and open up consideration of new ones. A popular variant of neural operators is the Fourier neural operator (FNO). Previous analysis proving universal operator approximation theorems for FNOs resorts to use of an unbounded number of Fourier modes, relying on intuition from traditional analysis of spectral methods. The present work challenges this point of view: (i) the work reduces FNO to its core essence, resulting in a minimal architecture termed the ``averaging neural operator'' (ANO); and (ii) analysis of the ANO shows that even this minimal ANO architecture benefits from universal approximation. This result is obtained based on only a spatial average as its only nonlocal ingredient (corresponding to retaining only a \emph{single} Fourier mode in the special case of the FNO). The analysis paves the way for a more systematic exploration of nonlocality, both through the development of new operator learning architectures and the analysis of existing and new architectures. Numerical results are presented which give insight into complexity issues related to the roles of channel width (embedding dimension) and number of Fourier modes.

Deep Operator Network Approximation Rates for Lipschitz Operators

Neural and spectral operator surrogates: unified construction and expression rate bounds

Operator Learning of Lipschitz Operators: An Information-Theoretic Perspective

Expression Rates of Neural Operators for Linear Elliptic PDEs in Polytopes

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equations

A discretization-invariant extension and analysis of some deep operator networks

An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constants

Optimal deep learning of holomorphic operators between Banach spaces

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

Neural Operator: Learning Maps Between Function Spaces

Best Approximation and Inverse Results for Neural Network Operators

Learning Lipschitz Operators with respect to Gaussian Measures with Near-Optimal Sample Complexity

Approximation and Generalization of DeepONets for Learning Operators Arising from a Class of Singularly Perturbed Problems

DNN Expression Rate Analysis of High-dimensional PDEs: Application to Option Pricing

Nonlocality and Nonlinearity Implies Universality in Operator Learning

Operator Learning: Algorithms and Analysis

Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations

On the Expressive Power of Neural Networks

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Computable Lipschitz Bounds for Deep Neural Networks