Abstract:The challenge of approximating functions in infinite-dimensional spaces from finite samples is widely regarded as formidable. We delve into the challenging problem of the numerical approximation of Sobolev-smooth functions defined on probability spaces. Our particular focus centers on the Wasserstein distance function, which serves as a relevant example. In contrast to the existing body of literature focused on approximating efficiently pointwise evaluations, we chart a new course to define functional approximants by adopting three machine learning-based approaches: 1. Solving a finite number of optimal transport problems and computing the corresponding Wasserstein potentials. 2. Employing empirical risk minimization with Tikhonov regularization in Wasserstein Sobolev spaces. 3. Addressing the problem through the saddle point formulation that characterizes the weak form of the Tikhonov functional's Euler-Lagrange equation. We furnish explicit and quantitative bounds on generalization errors for each of these solutions. We leverage the theory of metric Sobolev spaces and we combine it with techniques of optimal transport, variational calculus, and large deviation bounds. In our numerical implementation, we harness appropriately designed neural networks to serve as basis functions. These networks undergo training using diverse methodologies. This approach allows us to obtain approximating functions that can be rapidly evaluated after training. Our constructive solutions significantly enhance at equal accuracy the evaluation speed, surpassing that of state-of-the-art methods by several orders of magnitude. This allows evaluations over large datasets several times faster, including training, than traditional optimal transport algorithms. Our analytically designed deep learning architecture slightly outperforms the test error of state-of-the-art CNN architectures on datasets of images.

What problem does this paper attempt to address?

The main problem this paper attempts to address is how to efficiently approximate Sobolev smooth functions defined on the space of probability measures, particularly the Wasserstein distance function, given limited sample information. This problem is already very challenging in high-dimensional Euclidean spaces and becomes even more complex and difficult in infinite-dimensional metric spaces. Specifically, the authors focus on how to effectively approximate Sobolev smooth functions from a finite number of point evaluation information, with these functions being defined on the space of probability measures. They particularly focus on the Wasserstein distance function, which is an important example because the Wasserstein distance is the solution to the optimal transport problem. The paper proposes three machine learning-based methods to define function approximation: 1. **Solving a finite number of optimal transport problems and computing the corresponding Wasserstein potentials**: This method constructs the Wasserstein potential by solving a finite number of optimal transport problems, thereby approximating the Wasserstein distance. 2. **Using empirical risk minimization with Tikhonov regularization in the Wasserstein Sobolev space**: This method constructs function approximation by performing empirical risk minimization in the Wasserstein Sobolev space, combined with Tikhonov regularization techniques. 3. **Solving the Euler-Lagrange equation of the Tikhonov functional via a saddle point formulation**: This method constructs function approximation by formulating the Euler-Lagrange equation of the Tikhonov functional as a saddle point problem. Additionally, the authors provide explicit and quantitative generalization error bounds for each method and use well-designed neural networks as basis functions in the numerical implementation. These neural networks, once trained, can quickly evaluate functions, significantly improving evaluation speed while maintaining the same accuracy, outperforming existing methods by several orders of magnitude. This makes pairwise Wasserstein distance evaluation on large-scale datasets much faster than traditional optimal transport algorithms. In summary, this paper aims to address the problem of efficiently approximating Sobolev smooth functions on the space of probability measures by introducing new machine learning methods, particularly improving computational efficiency when dealing with large-scale data.

Approximation Theory, Computing, and Deep Learning on the Wasserstein Space

An Approximation Theory for Metric Space-Valued Functions With A View Towards Deep Learning

Variational Analysis in the Wasserstein Space

Approximation of splines in Wasserstein spaces

Neural approximation of Wasserstein distance via a universal architecture for symmetric and factorwise group invariant functions

Adaptive Approximation by Optimal Weighted Least-Squares Methods

A new method for determining Wasserstein 1 optimal transport maps from Kantorovich potentials, with deep learning applications

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

An Approximation Theory Framework for Measure-Transport Sampling Algorithms

A finite-dimensional approximation for partial differential equations on Wasserstein space

Optimal approximation of infinite-dimensional holomorphic functions

On the approximation of vector-valued functions by volume sampling

Accelerated First-order Methods on the Wasserstein Space for Bayesian Inference.

Deep learning based numerical approximation algorithms for stochastic partial differential equations and high-dimensional nonlinear filtering problems

Numerical Analysis on Neural Network Projected Schemes for Approximating One Dimensional Wasserstein Gradient Flows

Inference for Projection-Based Wasserstein Distances on Finite Spaces

Neural networks in non-metric spaces

Infinite-Variate $L^2$-Approximation with Nested Subspace Sampling

Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks