Adapted optimal transport between Gaussian processes in discrete time

Madhu Gunasingam,Ting-Kam Leonard Wong

2024-05-01

Abstract:We derive explicitly the adapted $2$-Wasserstein distance between non-degenerate Gaussian distributions on $\mathbb{R}^N$ and characterize the optimal bicausal coupling(s). This leads to an adapted version of the Bures-Wasserstein distance on the space of positive definite matrices.

Probability

What problem does this paper attempt to address?

Adapted Wasserstein distance between the laws of SDEs

Julio Backhoff-Veraguas,Sigrid Källblad,Benjamin A. Robinson

2024-10-14

Abstract:We consider the bicausal optimal transport problem between the laws of scalar time-homogeneous stochastic differential equations, and we establish the optimality of the synchronous coupling between these laws. The proof of this result is based on time-discretisation and reveals a novel connection between the synchronous coupling and the celebrated discrete-time Knothe--Rosenblatt rearrangement. We also prove a result on equality of topologies restricted to a certain subset of laws of continuous-time processes. We complement our main results with examples showing how the optimal coupling may change in path-dependent and multidimensional settings.

Probability
On optimal transport of matrix-valued measures

Yann Brenier,Dmitry Vorotnikov

DOI: https://doi.org/10.48550/arXiv.1808.05064

2019-07-15

Abstract:We suggest a new way of defining optimal transport of positive-semidefinite matrix-valued measures. It is inspired by a recent rendering of the incompressible Euler equations and related conservative systems as concave maximization problems. The main object of our attention is the Kantorovich-Bures metric space, which is a matricial analogue of the Wasserstein and Hellinger-Kantorovich metric spaces. We establish some topological, metric and geometric properties of this space, which includes the existence of the optimal transportation path.

Optimization and Control,Functional Analysis,Metric Geometry
Optimal transport for Gaussian mixture models

Yongxin Chen,Tryphon T Georgiou,Allen Tannenbaum

DOI: https://doi.org/10.1109/ACCESS.2018.2889838

IF: 3.9

IEEE Access

Abstract:We introduce an optimal mass transport framework on the space of Gaussian mixture models. These models are widely used in statistical inference. Specifically, we treat Gaussian mixture models as a submanifold of probability densities equipped with the Wasserstein metric. The topology induced by optimal transport is highly desirable and natural because, in contrast to total variation and other metrics, the Wasserstein metric is weakly continuous (i.e., convergence is equivalent to convergence of moments). Thus, our approach provides natural ways to compare, interpolate and average Gaussian mixture models. Moreover, the approach has low computational complexity. Different aspects of the framework are discussed and examples are presented for illustration purposes.
A Probabilistic View on the Adapted Wasserstein Distance

Mathias Beiglböck,Susanne Pflügl,Stefan Schrott

2024-06-28

Abstract:Causal optimal transport and adapted Wasserstein distance have applications in different fields from optimization to mathematical finance and machine learning. The goal of this article is to provide equivalent formulations of these concepts in classic probabilistic language. In particular, we prove a Skorokhod representation theorem for adapted weak convergence, reformulate the equivalence of stochastic processes using Markovian lifts, and give an expression for the adapted Wasserstein distance based on representing processes on a common stochastic basis.

Probability
On a general matrix-valued unbalanced optimal transport problem

Bowen Li,Jun Zou

2023-10-17

Abstract:We introduce a general class of transport distances ${\rm WB}_{\Lambda}$ over the space of positive semi-definite matrix-valued Radon measures $\mathcal{M}(\Omega,\mathbb{S}_+^n)$, called the weighted Wasserstein-Bures distance. Such a distance is defined via a generalized Benamou-Brenier formulation with a weighted action functional and an abstract matricial continuity equation, which leads to a convex optimization problem. Some recently proposed models, including the Kantorovich-Bures distance and the Wasserstein-Fisher-Rao distance, can naturally fit into ours. We give a complete characterization of the minimizer and explore the topological and geometrical properties of the space $(\mathcal{M}(\Omega,\mathbb{S}_+^n),{\rm WB}_{\Lambda})$. In particular, we show that $(\mathcal{M}(\Omega,\mathbb{S}_+^n),{\rm WB}_{\Lambda})$ is a complete geodesic space and exhibits a conic structure.

Numerical Analysis,Optimization and Control
Bicausal optimal transport for SDEs with irregular coefficients

Benjamin A. Robinson,Michaela Szölgyenyi

2024-09-27

Abstract:We solve constrained optimal transport problems in which the marginal laws are given by the laws of solutions of stochastic differential equations (SDEs). We consider SDEs with irregular coefficients, making only minimal regularity assumptions. We show that the so-called synchronous coupling is optimal among bicausal couplings, that is couplings that respect the flow of information encoded in the stochastic processes. Our results provide a method to numerically compute the adapted Wasserstein distance between laws of SDEs with irregular coefficients. We show that this can be applied to quantifying model uncertainty in stochastic optimisation problems. Moreover, we introduce a transformation-based semi-implicit numerical scheme and establish the first strong convergence result for SDEs with exponentially growing and discontinuous drift.

Probability,Numerical Analysis
Non-commutative Optimal Transport for semi-definite positive matrices

Augusto Gerolin,Nataliia Monina

2024-02-27

Abstract:We introduce the von Neumann entropy regularization of Unbalanced Non-commutative Optimal Transport, specifically Non-commutative Optimal Transport between semi-definite positive matrices (not necessarily with trace one). We prove the existence of a minimizer, compute the weak dual formulation and prove $\Gamma$-convergence results, demonstrating convergence to both Unbalanced Non-commutative Optimal Transport (as the Entropy-regularization parameter tends to zero) and von Neumann entropy regularized Non-commutative Optimal Transport problems (as the unbalanced penalty parameter tends to infinity). To draw an analogy to the Non-commutative case, we provide a concise introduction of the static formulation of Unbalanced Optimal Transport between positive measures and bounded cost

Optimization and Control,Functional Analysis
Optimal transport natural gradient for statistical manifolds with continuous sample space

Yifan Chen,Wuchen Li

2020-04-16

Abstract:We study the Wasserstein natural gradient in parametric statistical models with continuous sample spaces. Our approach is to pull back the $L^2$-Wasserstein metric tensor in the probability density space to a parameter space, equipping the latter with a positive definite metric tensor, under which it becomes a Riemannian manifold, named the Wasserstein statistical manifold. In general, it is not a totally geodesic sub-manifold of the density space, and therefore its geodesics will differ from the Wasserstein geodesics, except for the well-known Gaussian distribution case, a fact which can also be validated under our framework. We use the sub-manifold geometry to derive a gradient flow and natural gradient descent method in the parameter space. When parametrized densities lie in $\bR$, the induced metric tensor establishes an explicit formula. In optimization problems, we observe that the natural gradient descent outperforms the standard gradient descent when the Wasserstein distance is the objective function. In such a case, we prove that the resulting algorithm behaves similarly to the Newton method in the asymptotic regime. The proof calculates the exact Hessian formula for the Wasserstein distance, which further motivates another preconditioner for the optimization process. To the end, we present examples to illustrate the effectiveness of the natural gradient in several parametric statistical models, including the Gaussian measure, Gaussian mixture, Gamma distribution, and Laplace distribution.

Optimization and Control,Information Theory,Machine Learning,Statistics Theory
Optimal Transport and Wasserstein Barycenter for Radial Contoured Distributions

Keyu Chen,Yunxin Zhang

2024-11-13

Abstract:The optimal transport and Wasserstein barycenter of Gaussian distributions have been solved. In literature, the closed form formulas of the Monge map, the Wasserstein distance and the Wasserstein barycenter have been given. Moreover, when Gaussian distributions extend more generally to elliptical contoured distributions, similar results also hold true. In this case, Gaussian distributions are regarded as elliptical contoured distribution with generator function $e^{-x/2}$. However, there are few results about optimal transport for elliptical contoured distributions with different generator functions. In this paper, we degenerate elliptical contoured distributions to radial contoured distributions and study their optimal transport and prove their Wasserstein barycenter is still radial contoured. For general elliptical contoured distributions, we give two numerical counterexamples to show that the Wasserstein barycenter of elliptical contoured distributions does not have to be elliptical contoured.

Optimization and Control
Gromov-Wasserstein-like Distances in the Gaussian Mixture Models Space

Antoine Salmona,Julie Delon,Agnès Desolneux

2024-09-30

Abstract:The Gromov-Wasserstein (GW) distance is frequently used in machine learning to compare distributions across distinct metric spaces. Despite its utility, it remains computationally intensive, especially for large-scale problems. Recently, a novel Wasserstein distance specifically tailored for Gaussian mixture models (GMMs) and known as MW2 (mixture Wasserstein) has been introduced by several authors. In scenarios where data exhibit clustering, this approach simplifies to a small-scale discrete optimal transport problem, which complexity depends solely on the number of Gaussian components in the GMMs. This paper aims to incorporate invariance properties into MW2. This is done by introducing new Gromov-type distances, designed to be isometry-invariant in Euclidean spaces and applicable for comparing GMMs across different dimensional spaces. Our first contribution is the Mixture Gromov Wasserstein distance (MGW2), which can be viewed as a "Gromovized" version of MW2. This new distance has a straightforward discrete formulation, making it highly efficient for estimating distances between GMMs in practical applications. To facilitate the derivation of a transport plan between GMMs, we present a second distance, the Embedded Wasserstein distance (EW2). This distance turns out to be closely related to several recent alternatives to Gromov-Wasserstein. We show that EW2 can be adapted to derive a distance as well as optimal transportation plans between GMMs. We demonstrate the efficiency of these newly proposed distances on medium to large-scale problems, including shape matching and hyperspectral image color transfer.

Machine Learning,Computer Vision and Pattern Recognition
Optimal Transport for Domain Adaptation through Gaussian Mixture Models

Eduardo Fernandes Montesuma,Fred Maurice Ngolè Mboula,Antoine Souloumiac

2024-03-18

Abstract:In this paper we explore domain adaptation through optimal transport. We propose a novel approach, where we model the data distributions through Gaussian mixture models. This strategy allows us to solve continuous optimal transport through an equivalent discrete problem. The optimal transport solution gives us a matching between source and target domain mixture components. From this matching, we can map data points between domains, or transfer the labels from the source domain components towards the target domain. We experiment with 2 domain adaptation benchmarks in fault diagnosis, showing that our methods have state-of-the-art performance.

Machine Learning,Artificial Intelligence
Projected Langevin dynamics and a gradient flow for entropic optimal transport

Giovanni Conforti,Daniel Lacker,Soumik Pal

2023-09-16

Abstract:The classical (overdamped) Langevin dynamics provide a natural algorithm for sampling from its invariant measure, which uniquely minimizes an energy functional over the space of probability measures, and which concentrates around the minimizer(s) of the associated potential when the noise parameter is small. We introduce analogous diffusion dynamics that sample from an entropy-regularized optimal transport, which uniquely minimizes the same energy functional but constrained to the set $\Pi(\mu,\nu)$ of couplings of two given marginal probability measures $\mu$ and $\nu$ on $\mathbb{R}^d$, and which concentrates around the optimal transport coupling(s) for small regularization parameter. More specifically, our process satisfies two key properties: First, the law of the solution at each time stays in $\Pi(\mu,\nu)$ if it is initialized there. Second, the long-time limit is the unique solution of an entropic optimal transport problem. In addition, we show by means of a new log-Sobolev-type inequality that the convergence holds exponentially fast, for sufficiently large regularization parameter and for a class of marginals which strictly includes all strongly log-concave measures. By studying the induced Wasserstein geometry of the submanifold $\Pi(\mu,\nu)$, we argue that the SDE can be viewed as a Wasserstein gradient flow on this space of couplings, at least when $d=1$, and we identify a conjectural gradient flow for $d \ge 2$. The main technical difficulties stems from the appearance of conditional expectation terms which serve to constrain the dynamics to $\Pi(\mu,\nu)$.

Probability,Analysis of PDEs,Machine Learning
Optimal 1-Wasserstein Distance for WGANs

Arthur Stéphanovitch,Ugo Tanielian,Benoît Cadre,Nicolas Klutchnikoff,Gérard Biau

2023-10-05

Abstract:The mathematical forces at work behind Generative Adversarial Networks raise challenging theoretical issues. Motivated by the important question of characterizing the geometrical properties of the generated distributions, we provide a thorough analysis of Wasserstein GANs (WGANs) in both the finite sample and asymptotic regimes. We study the specific case where the latent space is univariate and derive results valid regardless of the dimension of the output space. We show in particular that for a fixed sample size, the optimal WGANs are closely linked with connected paths minimizing the sum of the squared Euclidean distances between the sample points. We also highlight the fact that WGANs are able to approach (for the 1-Wasserstein distance) the target distribution as the sample size tends to infinity, at a given convergence rate and provided the family of generative Lipschitz functions grows appropriately. We derive in passing new results on optimal transport theory in the semi-discrete setting.

Machine Learning,Statistics Theory
Rate of convergence of the smoothed empirical Wasserstein distance

Adam Block,Zeyu Jia,Yury Polyanskiy,Alexander Rakhlin

2024-08-16

Abstract:Consider an empirical measure $\mathbb{P}_n$ induced by $n$ iid samples from a $d$-dimensional $K$-subgaussian distribution $\mathbb{P}$ and let $\gamma = N(0,\sigma^2 I_d)$ be the isotropic Gaussian measure. We study the speed of convergence of the smoothed Wasserstein distance $W_2(\mathbb{P}_n * \gamma, \mathbb{P}*\gamma) = n^{-\alpha + o(1)}$ with $*$ being the convolution of measures. For $K<\sigma$ and in any dimension $d\ge 1$ we show that $\alpha = {1\over2}$. For $K>\sigma$ in dimension $d=1$ we show that the rate is slower and is given by $\alpha = {(\sigma^2 + K^2)^2\over 4 (\sigma^4 + K^4)} < 1/2$. This resolves several open problems in [GGNWP20], and in particular precisely identifies the amount of smoothing $\sigma$ needed to obtain a parametric rate. In addition, for any $d$-dimensional $K$-subgaussian distribution $\mathbb{P}$, we also establish that $D_{KL}(\mathbb{P}_n * \gamma \|\mathbb{P}*\gamma)$ has rate $O(1/n)$ for $K<\sigma$ but only slows down to $O({(\log n)^{d+1}\over n})$ for $K>\sigma$. The surprising difference of the behavior of $W_2^2$ and KL implies the failure of $T_{2}$-transportation inequality when $\sigma < K$. Consequently, it follows that for $K>\sigma$ the log-Sobolev inequality (LSI) for the Gaussian mixture $\mathbb{P} * N(0, \sigma^{2})$ cannot hold. This closes an open problem in [WW+16], who established the LSI under the condition $K<\sigma$ and asked if their bound can be improved.

Probability,Information Theory,Statistics Theory
Fast Entropic Regularized Optimal Transport Using Semidiscrete Cost Approximation

Evgeny Tenetov,Gershon Wolansky,Ron Kimmel

DOI: https://doi.org/10.1137/17m1162925

IF: 2.968

2018-01-01

SIAM Journal on Scientific Computing

Abstract:The optimal transportation theory was successfully applied to different tasks on geometric domains as images and triangle meshes. In these applications the transport problem is defined on a Riemannian manifold with geodesic distance $d(x,y)$. Usually, the cost function used is the geodesic distance $d$ or the squared geodesic distance $d^2$. These choices result in the 1-Wasserstein distance, also known as the earth mover's distance (EMD), or the 2-Wasserstein distance. The entropy regularized optimal transport problem can be solved using the Bregman projection algorithm. This algorithm can be implemented using only matrix multiplications of matrix $\exp(-C/varepsilon)$ (pointwise exponent) and pointwise vector multiplications, where $C$ is a cost matrix, and $\varepsilon$ is the regularization parameter. In this paper, we obtain a low-rank decomposition of this matrix and exploit it to accelerate the Bregman projection algorithm. Our low-rank decomposition is based on the semidiscrete approximation of the cost function, which is valid for a large family of cost functions, including $d^p (x, y)$, where $p \geq 1$. Our method requires the calculation of only a small portion of the distances.

mathematics, applied
Gaussian mixtures closest to a given measure via optimal transport

Jean-Bernard Lasserre

2024-04-30

Abstract:Given a determinate (multivariate) probability measure $\mu$, we characterize Gaussian mixtures $\nu\_\phi$ which minimize the Wasserstein distance $W\_2(\mu,\nu\_\phi)$ to $\mu$ when the mixing probability measure $\phi$ on the parameters $(m,\Sigma)$ of the Gaussians is supported on a compact set $S$.(i) We first show that such mixtures are optimal solutions of a particular optimal transport (OT) problem where the marginal $\nu\_{\phi}$ of the OT problem is also unknown via the mixing measure variable $\phi$. Next (ii) by using a well-known specific property of Gaussian measures, this optimal transport is then viewed as a Generalized Moment Problem (GMP) and if the set $S$ of mixture parameters $(m,\Sigma)$ is a basic compact semi-algebraic set, we provide a "mesh-free" numerical scheme to approximate as closely as desired the optimal distance by solving a hierarchy of semidefinite relaxations of increasing size. In particular, we neither assume that the mixing measure is finitely supported nor that the variance is the same for all components. If the original measure $\mu$ is not a Gaussian mixture with parameters $(m,\Sigma)\in S$, then a strictly positive distance is detected at a finite step of the hierarchy. If the original measure $\mu$ is a Gaussian mixture with parameters $(m,\Sigma)\in S$, then all semidefinite relaxations of the hierarchy have same zero optimal value. Moreover if the mixing measure is atomic with finite support, its components can sometimes be extracted from an optimal solution at some semidefinite relaxation of the hierarchy when Curto & Fialkow's flatness condition holds for some moment matrix.

Optimization and Control,Statistics Theory
Generalized optimal transport with singular sources

Jan Maas,Martin Rumpf,Stefan Simon

DOI: https://doi.org/10.48550/arXiv.1607.01186

2016-07-05

Numerical Analysis

Abstract:We present a generalized optimal transport model in which the mass-preserving constraint for the $L^2$-Wasserstein distance is relaxed by introducing a source term in the continuity equation. The source term is also incorporated in the path energy by means of its squared $L^2$-norm in time of a functional with linear growth in space. This extension of the original transport model enables local density modulation, which is a desirable feature in applications such as image warping and blending. A key advantage of the use of a functional with linear growth in space is that it allows for singular sources and sinks, which can be supported on points or lines. On a technical level, the $L^2$-norm in time ensures a disintegration of the source in time, which we use to obtain the well-posedness of the model and the existence of geodesic paths. Furthermore, a numerical scheme based on the proximal splitting approach (Papadakis et al., 2014) is presented. We compare our model with the corresponding model involving the $L^2(L^2)$-norm of the source, which merges the metamorphosis approach and the optimal transport approaches in imaging. Selected numerical test cases show strikingly different behaviour.
A new method for determining Wasserstein 1 optimal transport maps from Kantorovich potentials, with deep learning applications

Tristan Milne,Étienne Bilocq,Adrian Nachman

DOI: https://doi.org/10.48550/arXiv.2211.00820

2022-11-02

Abstract:Wasserstein 1 optimal transport maps provide a natural correspondence between points from two probability distributions, $\mu$ and $\nu$, which is useful in many applications. Available algorithms for computing these maps do not appear to scale well to high dimensions. In deep learning applications, efficient algorithms have been developed for approximating solutions of the dual problem, known as Kantorovich potentials, using neural networks (e.g. [Gulrajani et al., 2017]). Importantly, such algorithms work well in high dimensions. In this paper we present an approach towards computing Wasserstein 1 optimal transport maps that relies only on Kantorovich potentials. In general, a Wasserstein 1 optimal transport map is not unique and is not computable from a potential alone. Our main result is to prove that if $\mu$ has a density and $\nu$ is supported on a submanifold of codimension at least 2, an optimal transport map is unique and can be written explicitly in terms of a potential. These assumptions are natural in many image processing contexts and other applications. When the Kantorovich potential is only known approximately, our result motivates an iterative procedure wherein data is moved in optimal directions and with the correct average displacement. Since this provides an approach for transforming one distribution to another, it can be used as a multipurpose algorithm for various transport problems; we demonstrate through several proof of concept experiments that this algorithm successfully performs various imaging tasks, such as denoising, generation, translation and deblurring, which normally require specialized techniques.

Optimization and Control,Computer Vision and Pattern Recognition,Machine Learning,Neural and Evolutionary Computing
Optimal Transport for Kernel Gaussian Mixture Models

Jung Hun Oh,Rena Elkin,Anish Kumar Simhal,Jiening Zhu,Joseph O Deasy,Allen Tannenbaum

2023-10-28

Abstract:The Wasserstein distance from optimal mass transport (OMT) is a powerful mathematical tool with numerous applications that provides a natural measure of the distance between two probability distributions. Several methods to incorporate OMT into widely used probabilistic models, such as Gaussian or Gaussian mixture, have been developed to enhance the capability of modeling complex multimodal densities of real datasets. However, very few studies have explored the OMT problems in a reproducing kernel Hilbert space (RKHS), wherein the kernel trick is utilized to avoid the need to explicitly map input data into a high-dimensional feature space. In the current study, we propose a Wasserstein-type metric to compute the distance between two Gaussian mixtures in a RKHS via the kernel trick, i.e., kernel Gaussian mixture models.

Machine Learning
Fast Dual Subgradient Optimization of the Integrated Transportation Distance Between Stochastic Kernels

Zhengqi Lin,Andrzej Ruszczynski

2023-12-03

Abstract:A generalization of the Wasserstein metric, the integrated transportation distance, establishes a novel distance between probability kernels of Markov systems. This metric serves as the foundation for an efficient approximation technique, enabling the replacement of the original system's kernel with a kernel with a discrete support of limited cardinality. To facilitate practical implementation, we present a specialized dual algorithm capable of constructing these approximate kernels quickly and efficiently, without requiring computationally expensive matrix operations. Finally, we demonstrate the efficacy of our method through several illustrative examples, showcasing its utility in practical scenarios. This advancement offers new possibilities for the streamlined analysis and manipulation of stochastic systems represented by kernels.

Machine Learning,Optimization and Control

Adapted optimal transport between Gaussian processes in discrete time

Adapted Wasserstein distance between the laws of SDEs

On optimal transport of matrix-valued measures

Optimal transport for Gaussian mixture models

A Probabilistic View on the Adapted Wasserstein Distance

On a general matrix-valued unbalanced optimal transport problem

Bicausal optimal transport for SDEs with irregular coefficients

Non-commutative Optimal Transport for semi-definite positive matrices

Optimal transport natural gradient for statistical manifolds with continuous sample space

Optimal Transport and Wasserstein Barycenter for Radial Contoured Distributions

Gromov-Wasserstein-like Distances in the Gaussian Mixture Models Space

Optimal Transport for Domain Adaptation through Gaussian Mixture Models

Projected Langevin dynamics and a gradient flow for entropic optimal transport

Optimal 1-Wasserstein Distance for WGANs

Rate of convergence of the smoothed empirical Wasserstein distance

Fast Entropic Regularized Optimal Transport Using Semidiscrete Cost Approximation

Gaussian mixtures closest to a given measure via optimal transport

Generalized optimal transport with singular sources

A new method for determining Wasserstein 1 optimal transport maps from Kantorovich potentials, with deep learning applications

Optimal Transport for Kernel Gaussian Mixture Models

Fast Dual Subgradient Optimization of the Integrated Transportation Distance Between Stochastic Kernels