Abstract:Learning distance functions between complex objects, such as the Wasserstein distance to compare point sets, is a common goal in machine learning applications. However, functions on such complex objects (e.g., point sets and graphs) are often required to be invariant to a wide variety of group actions e.g. permutation or rigid transformation. Therefore, continuous and symmetric product functions (such as distance functions) on such complex objects must also be invariant to the product of such group actions. We call these functions symmetric and factor-wise group invariant (or SFGI functions in short). In this paper, we first present a general neural network architecture for approximating SFGI functions. The main contribution of this paper combines this general neural network with a sketching idea to develop a specific and efficient neural network which can approximate the $p$-th Wasserstein distance between point sets. Very importantly, the required model complexity is independent of the sizes of input point sets. On the theoretical front, to the best of our knowledge, this is the first result showing that there exists a neural network with the capacity to approximate Wasserstein distance with bounded model complexity. Our work provides an interesting integration of sketching ideas for geometric problems with universal approximation of symmetric functions. On the empirical front, we present a range of results showing that our newly proposed neural network architecture performs comparatively or better than other models (including a SOTA Siamese Autoencoder based approach). In particular, our neural network generalizes significantly better and trains much faster than the SOTA Siamese AE. Finally, this line of investigation could be useful in exploring effective neural network design for solving a broad range of geometric optimization problems (e.g., $k$-means in a metric space).

Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machines

Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks

Decomposition of Equivariant Maps via Invariant Maps: Application to Universal Approximation under Symmetry

Learning Functions: When Is Deep Better Than Shallow

Universality of deep convolutional neural networks

Algebraic geometrical methods for hierarchical learning machines

Universal Approximation Property of Quantum Machine Learning Models in Quantum-Enhanced Feature Spaces

Deep Narrow Boltzmann Machines are Universal Approximators

Universal Approximation Theorem for Neural Networks

Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networks

Deterministic equivalent and error universality of deep random features learning *

Deep vs. shallow networks : An approximation theory perspective

Universality and Optimality of Structured Deep Kernel Networks

Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

Neural approximation of Wasserstein distance via a universal architecture for symmetric and factorwise group invariant functions

Universal approximation with complex-valued deep narrow neural networks

A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations

A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks

Building Efficient Deep Neural Networks With Unitary Group Convolutions