Abstract:The challenge of approximating functions in infinite-dimensional spaces from finite samples is widely regarded as formidable. We delve into the challenging problem of the numerical approximation of Sobolev-smooth functions defined on probability spaces. Our particular focus centers on the Wasserstein distance function, which serves as a relevant example. In contrast to the existing body of literature focused on approximating efficiently pointwise evaluations, we chart a new course to define functional approximants by adopting three machine learning-based approaches: 1. Solving a finite number of optimal transport problems and computing the corresponding Wasserstein potentials. 2. Employing empirical risk minimization with Tikhonov regularization in Wasserstein Sobolev spaces. 3. Addressing the problem through the saddle point formulation that characterizes the weak form of the Tikhonov functional's Euler-Lagrange equation. We furnish explicit and quantitative bounds on generalization errors for each of these solutions. We leverage the theory of metric Sobolev spaces and we combine it with techniques of optimal transport, variational calculus, and large deviation bounds. In our numerical implementation, we harness appropriately designed neural networks to serve as basis functions. These networks undergo training using diverse methodologies. This approach allows us to obtain approximating functions that can be rapidly evaluated after training. Our constructive solutions significantly enhance at equal accuracy the evaluation speed, surpassing that of state-of-the-art methods by several orders of magnitude. This allows evaluations over large datasets several times faster, including training, than traditional optimal transport algorithms. Our analytically designed deep learning architecture slightly outperforms the test error of state-of-the-art CNN architectures on datasets of images.

An Approximation Theory for Metric Space-Valued Functions With A View Towards Deep Learning

Neural networks in non-metric spaces

Approximation Theory, Computing, and Deep Learning on the Wasserstein Space

Geometric and arithmetic aspects of approximation vectors

An Approximation Theory Framework for Measure-Transport Sampling Algorithms

Projection Methods for Operator Learning and Universal Approximation

Approximation in Hölder Spaces

Universal approximation property of Banach space-valued random feature models including random neural networks

Metric Approximation of Set-Valued Functions of Bounded Variation by Integral Operators

Approximation of discontinuous functions by positive linear operators. A probabilistic approach

Analysis of Approximation by Linear Operators on VariableLρp(·)Spaces and Applications in Learning Theory

On the approximation of vector-valued functions by volume sampling

Global universal approximation of functional input maps on weighted spaces

Discrete Approximation of Optimal Transport on Compact Spaces

Random Vector Functional Link Networks for Function Approximation on Manifolds

Universal approximation on non-geometric rough paths and applications to financial derivatives pricing

Linearly Approximatable Functions

Universal approximation theorems for continuous functions of càdlàg paths and Lévy-type signature models

A Theory of Interpretable Approximations

Deep Learning via Dynamical Systems: An Approximation Perspective

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks