A Gradient Descent Perspective on Sinkhorn

Flavien Léger

DOI: https://doi.org/10.1007/s00245-020-09697-w

2020-07-01

Abstract:We present a new perspective on the popular Sinkhorn algorithm, showing that it can be seen as a Bregman gradient descent (mirror descent) of a relative entropy (Kullback–Leibler divergence). This viewpoint implies a new sublinear convergence rate with a robust constant.

mathematics, applied

What problem does this paper attempt to address?

Sinkhorn Barycenter via Functional Gradient Descent

Zebang Shen,Zhenfu Wang,Alejandro Ribeiro,Hamed Hassani

2020-01-01

Abstract: In this paper, we consider the problem of computing the barycenter of a set of probability distributions under the Sinkhorn divergence. This problem has recently found applications across various domains, including graphics, learning, and vision, as it provides a meaningful mechanism to aggregate knowledge. Unlike previous approaches which directly operate in the space of probability measures, we recast the Sinkhorn barycenter problem as an instance of unconstrained functional optimization and develop a novel functional gradient descent method named Sinkhorn Descent (SD). We prove that SD converges to a stationary point at a sublinear rate, and under reasonable assumptions, we further show that it asymptotically finds a global minimizer of the Sinkhorn barycenter problem. Moreover, by providing a mean-field analysis, we show that SD preserves the weak convergence of empirical measures. Importantly, the computational complexity of SD scales linearly in the dimension $d$ and we demonstrate its scalability by solving a $100$-dimensional Sinkhorn barycenter problem.
Sinkhorn Flow: A Continuous-Time Framework for Understanding and Generalizing the Sinkhorn Algorithm

Mohammad Reza Karimi,Ya-Ping Hsieh,Andreas Krause

2023-11-28

Abstract:Many problems in machine learning can be formulated as solving entropy-regularized optimal transport on the space of probability measures. The canonical approach involves the Sinkhorn iterates, renowned for their rich mathematical properties. Recently, the Sinkhorn algorithm has been recast within the mirror descent framework, thus benefiting from classical optimization theory insights. Here, we build upon this result by introducing a continuous-time analogue of the Sinkhorn algorithm. This perspective allows us to derive novel variants of Sinkhorn schemes that are robust to noise and bias. Moreover, our continuous-time dynamics not only generalize but also offer a unified perspective on several recently discovered dynamics in machine learning and mathematics, such as the "Wasserstein mirror flow" of (Deb et al. 2023) or the "mean-field Schrödinger equation" of (Claisse et al. 2023).

Machine Learning,Probability
Non-asymptotic convergence bounds for Sinkhorn iterates and their gradients: a coupling approach

Giacomo Greco,Maxence Noble,Giovanni Conforti,Alain Durmus

2023-06-27

Abstract:Computational optimal transport (OT) has recently emerged as a powerful framework with applications in various fields. In this paper we focus on a relaxation of the original OT problem, the entropic OT problem, which allows to implement efficient and practical algorithmic solutions, even in high dimensional settings. This formulation, also known as the Schrödinger Bridge problem, notably connects with Stochastic Optimal Control (SOC) and can be solved with the popular Sinkhorn algorithm. In the case of discrete-state spaces, this algorithm is known to have exponential convergence; however, achieving a similar rate of convergence in a more general setting is still an active area of research. In this work, we analyze the convergence of the Sinkhorn algorithm for probability measures defined on the $d$-dimensional torus $\mathbb{T}_L^d$, that admit densities with respect to the Haar measure of $\mathbb{T}_L^d$. In particular, we prove pointwise exponential convergence of Sinkhorn iterates and their gradient. Our proof relies on the connection between these iterates and the evolution along the Hamilton-Jacobi-Bellman equations of value functions obtained from SOC-problems. Our approach is novel in that it is purely probabilistic and relies on coupling by reflection techniques for controlled diffusions on the torus.

Probability,Optimization and Control,Machine Learning
Sinkhorn Natural Gradient for Generative Models

Zebang Shen,Zhenfu Wang,Alejandro Ribeiro,Hamed Hassani

2020-01-01

Abstract: We consider the problem of minimizing a functional over a parametric family of probability measures, where the parameterization is characterized via a push-forward structure. An important application of this problem is in training generative adversarial networks. In this regard, we propose a novel Sinkhorn Natural Gradient (SiNG) algorithm which acts as a steepest descent method on the probability space endowed with the Sinkhorn divergence. We show that the Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit expression and can be evaluated accurately in complexity that scales logarithmically with respect to the desired accuracy. This is in sharp contrast to existing natural gradient methods that can only be carried out approximately. Moreover, in practical applications when only Monte-Carlo type integration is available, we design an empirical estimator for SIM and provide the stability analysis. In our experiments, we quantitatively compare SiNG with state-of-the-art SGD-type solvers on generative tasks to demonstrate its efficiency and efficacy of our method.
On the geometry and dynamical formulation of the Sinkhorn algorithm for optimal transport

Klas Modin

2024-01-24

Abstract:The Sinkhorn algorithm is a numerical method for the solution of optimal transport problems. Here, I give a brief survey of this algorithm, with a strong emphasis on its geometric origin: it is natural to view it as a discretization, by standard methods, of a non-linear integral equation. In the appendix, I also provide a short summary of an early result of Beurling on product measures, directly related to the Sinkhorn algorithm.

Numerical Analysis,Differential Geometry
Accelerating Sinkhorn Algorithm with Sparse Newton Iterations

Xun Tang,Michael Shavlovsky,Holakou Rahmanian,Elisa Tardini,Kiran Koshy Thekumparampil,Tesi Xiao,Lexing Ying

2024-01-21

Abstract:Computing the optimal transport distance between statistical distributions is a fundamental task in machine learning. One remarkable recent advancement is entropic regularization and the Sinkhorn algorithm, which utilizes only matrix scaling and guarantees an approximated solution with near-linear runtime. Despite the success of the Sinkhorn algorithm, its runtime may still be slow due to the potentially large number of iterations needed for convergence. To achieve possibly super-exponential convergence, we present Sinkhorn-Newton-Sparse (SNS), an extension to the Sinkhorn algorithm, by introducing early stopping for the matrix scaling steps and a second stage featuring a Newton-type subroutine. Adopting the variational viewpoint that the Sinkhorn algorithm maximizes a concave Lyapunov potential, we offer the insight that the Hessian matrix of the potential function is approximately sparse. Sparsification of the Hessian results in a fast $O(n^2)$ per-iteration complexity, the same as the Sinkhorn algorithm. In terms of total iteration count, we observe that the SNS algorithm converges orders of magnitude faster across a wide range of practical cases, including optimal transportation between empirical distributions and calculating the Wasserstein $W_1, W_2$ distance of discretized densities. The empirical performance is corroborated by a rigorous bound on the approximate sparsity of the Hessian matrix.

Optimization and Control,Machine Learning
Re-basin via implicit Sinkhorn differentiation

Fidel A. Guerrero Peña,Heitor Rapela Medeiros,Thomas Dubail,Masih Aminbeidokhti,Eric Granger,Marco Pedersoli

DOI: https://doi.org/10.48550/arXiv.2212.12042

2022-12-23

Abstract:The recent emergence of new algorithms for permuting models into functionally equivalent regions of the solution space has shed some light on the complexity of error surfaces, and some promising properties like mode connectivity. However, finding the right permutation is challenging, and current optimization techniques are not differentiable, which makes it difficult to integrate into a gradient-based optimization, and often leads to sub-optimal solutions. In this paper, we propose a Sinkhorn re-basin network with the ability to obtain the transportation plan that better suits a given objective. Unlike the current state-of-art, our method is differentiable and, therefore, easy to adapt to any task within the deep learning domain. Furthermore, we show the advantage of our re-basin method by proposing a new cost function that allows performing incremental learning by exploiting the linear mode connectivity property. The benefit of our method is compared against similar approaches from the literature, under several conditions for both optimal transport finding and linear mode connectivity. The effectiveness of our continual learning method based on re-basin is also shown for several common benchmark datasets, providing experimental results that are competitive with state-of-art results from the literature.

Computer Vision and Pattern Recognition
Mirror and Preconditioned Gradient Descent in Wasserstein Space

Clément Bonet,Théo Uscidda,Adam David,Pierre-Cyril Aubin-Frankowski,Anna Korba

2024-06-13

Abstract:As the problem of minimizing functionals on the Wasserstein space encompasses many applications in machine learning, different optimization algorithms on $\mathbb{R}^d$ have received their counterpart analog on the Wasserstein space. We focus here on lifting two explicit algorithms: mirror descent and preconditioned gradient descent. These algorithms have been introduced to better capture the geometry of the function to minimize and are provably convergent under appropriate (namely relative) smoothness and convexity conditions. Adapting these notions to the Wasserstein space, we prove guarantees of convergence of some Wasserstein-gradient-based discrete-time schemes for new pairings of objective functionals and regularizers. The difficulty here is to carefully select along which curves the functionals should be smooth and convex. We illustrate the advantages of adapting the geometry induced by the regularizer on ill-conditioned optimization tasks, and showcase the improvement of choosing different discrepancies and geometries in a computational biology task of aligning single-cells.

Optimization and Control,Machine Learning
The Information Geometry of Mirror Descent

Garvesh Raskutti,Sayan Mukherjee

DOI: https://doi.org/10.1109/tit.2015.2388583

IF: 2.5

2015-03-01

IEEE Transactions on Information Theory

Abstract:We prove the equivalence of two online learning algorithms: 1) mirror descent and 2) natural gradient descent. Both mirror descent and natural gradient descent are generalizations of online gradient descent when the parameter of interest lies on a non-Euclidean manifold. Natural gradient descent selects the steepest descent along a Riemannian manifold by multiplying the standard gradient by the inverse of the metric tensor. Mirror descent induces non-Euclidean structure by solving iterative optimization problems using different proximity functions. In this paper, we prove that mirror descent induced by Bregman divergence proximity functions is equivalent to the natural gradient descent algorithm on the dual Riemannian manifold. We use techniques from convex analysis and connections between Riemannian manifolds, Bregman divergences, and convexity to prove this result. This equivalence between natural gradient descent and mirror descent, implies that: 1) mirror descent is the steepest descent direction along the Riemannian manifold corresponding to the choice of Bregman divergence and 2) mirror descent with log-likelihood loss applied to parameter estimation in exponential families asymptotically achieves the classical Cramér–Rao lower bound.

computer science, information systems,engineering, electrical & electronic
Gradient is All You Need?

Konstantin Riedl,Timo Klock,Carina Geldhauser,Massimo Fornasier

2023-06-16

Abstract:In this paper we provide a novel analytical perspective on the theoretical understanding of gradient-based learning algorithms by interpreting consensus-based optimization (CBO), a recently proposed multi-particle derivative-free optimization method, as a stochastic relaxation of gradient descent. Remarkably, we observe that through communication of the particles, CBO exhibits a stochastic gradient descent (SGD)-like behavior despite solely relying on evaluations of the objective function. The fundamental value of such link between CBO and SGD lies in the fact that CBO is provably globally convergent to global minimizers for ample classes of nonsmooth and nonconvex objective functions, hence, on the one side, offering a novel explanation for the success of stochastic relaxations of gradient descent. On the other side, contrary to the conventional wisdom for which zero-order methods ought to be inefficient or not to possess generalization abilities, our results unveil an intrinsic gradient descent nature of such heuristics. This viewpoint furthermore complements previous insights into the working principles of CBO, which describe the dynamics in the mean-field limit through a nonlinear nonlocal partial differential equation that allows to alleviate complexities of the nonconvex function landscape. Our proofs leverage a completely nonsmooth analysis, which combines a novel quantitative version of the Laplace principle (log-sum-exp trick) and the minimizing movement scheme (proximal iteration). In doing so, we furnish useful and precise insights that explain how stochastic perturbations of gradient descent overcome energy barriers and reach deep levels of nonconvex functions. Instructive numerical illustrations support the provided theoretical insights.

Machine Learning,Numerical Analysis,Optimization and Control
Rethinking Initialization of the Sinkhorn Algorithm

James Thornton,Marco Cuturi

DOI: https://doi.org/10.48550/arXiv.2206.07630

2023-04-05

Abstract:While the optimal transport (OT) problem was originally formulated as a linear program, the addition of entropic regularization has proven beneficial both computationally and statistically, for many applications. The Sinkhorn fixed-point algorithm is the most popular approach to solve this regularized problem, and, as a result, multiple attempts have been made to reduce its runtime using, e.g., annealing in the regularization parameter, momentum or acceleration. The premise of this work is that initialization of the Sinkhorn algorithm has received comparatively little attention, possibly due to two preconceptions: since the regularized OT problem is convex, it may not be worth crafting a good initialization, since any is guaranteed to work; secondly, because the outputs of the Sinkhorn algorithm are often unrolled in end-to-end pipelines, a data-dependent initialization would bias Jacobian computations. We challenge this conventional wisdom, and show that data-dependent initializers result in dramatic speed-ups, with no effect on differentiability as long as implicit differentiation is used. Our initializations rely on closed-forms for exact or approximate OT solutions that are known in the 1D, Gaussian or GMM settings. They can be used with minimal tuning, and result in consistent speed-ups for a wide variety of OT problems.

Machine Learning
A Mirror Descent Perspective of Smoothed Sign Descent

Shuyang Wang,Diego Klabjan

2024-10-18

Abstract:Recent work by Woodworth et al. (2020) shows that the optimization dynamics of gradient descent for overparameterized problems can be viewed as low-dimensional dual dynamics induced by a mirror map, explaining the implicit regularization phenomenon from the mirror descent perspective. However, the methodology does not apply to algorithms where update directions deviate from true gradients, such as ADAM. We use the mirror descent framework to study the dynamics of smoothed sign descent with a stability constant $\varepsilon$ for regression problems. We propose a mirror map that establishes equivalence to dual dynamics under some assumptions. By studying dual dynamics, we characterize the convergent solution as an approximate KKT point of minimizing a Bregman divergence style function, and show the benefit of tuning the stability constant $\varepsilon$ to reduce the KKT error.

Machine Learning,Optimization and Control
A semiconcavity approach to stability of entropic plans and exponential convergence of Sinkhorn's algorithm

Alberto Chiarini,Giovanni Conforti,Giacomo Greco,Luca Tamanini

2024-12-12

Abstract:We study stability of optimizers and convergence of Sinkhorn's algorithm in the framework of entropic optimal transport. We show entropic stability for optimal plans in terms of the Wasserstein distance between their marginals under a semiconcavity assumption on the sum of the cost and one of the two entropic potentials. When employed in the analysis of Sinkhorn's algorithm, this result gives a natural sufficient condition for its exponential convergence, which does not require the ground cost to be bounded. By controlling from above the Hessians of Sinkhorn potentials in examples of interest, we obtain new exponential convergence results. For instance, for the first time we obtain exponential convergence for log-concave marginals and quadratic costs for all values of the regularization parameter. Moreover, the convergence rate has a linear dependence on the regularization: this behavior is sharp and had only been previously obtained for compact distributions <a class="link-https" data-arxiv-id="2407.01202" href="https://arxiv.org/abs/2407.01202">arXiv:2407.01202</a>. Other interesting new applications include subspace elastic costs [Cuturi et al. PMLR 202(2023)], weakly log-concave marginals, marginals with light tails, where, under reinforced assumptions, we manage to improve the rates obtained in <a class="link-https" data-arxiv-id="2311.04041" href="https://arxiv.org/abs/2311.04041">arXiv:2311.04041</a>, the case of unbounded Lipschitz costs, and compact Riemannian manifolds.

Probability,Optimization and Control,Machine Learning
Linear Time Sinkhorn Divergences using Positive Features

Meyer Scetbon,Marco Cuturi

DOI: https://doi.org/10.48550/arXiv.2006.07057

IF: 5.414

2020-06-12

Machine Learning

Abstract:Although Sinkhorn divergences are now routinely used in data sciences to compare probability distributions, the computational effort required to compute them remains expensive, growing in general quadratically in the size $n$ of the support of these distributions. Indeed, solving optimal transport (OT) with an entropic regularization requires computing a $n\times n$ kernel matrix (the neg-exponential of a $n\times n$ pairwise ground cost matrix) that is repeatedly applied to a vector. We propose to use instead ground costs of the form $c(x,y)=-\log\dotp{\varphi(x)}{\varphi(y)}$ where $\varphi$ is a map from the ground space onto the positive orthant $\RR^r_+$, with $r\ll n$. This choice yields, equivalently, a kernel $k(x,y)=\dotp{\varphi(x)}{\varphi(y)}$, and ensures that the cost of Sinkhorn iterations scales as $O(nr)$. We show that usual cost functions can be approximated using this form. Additionaly, we take advantage of the fact that our approach yields approximation that remain fully differentiable with respect to input distributions, as opposed to previously proposed adaptive low-rank approximations of the kernel matrix, to train a faster variant of OT-GAN \cite{salimans2018improving}.
Annealed Sinkhorn for Optimal Transport: convergence, regularization path and debiasing

Lénaïc Chizat

2024-08-21

Abstract:Sinkhorn's algorithm is a method of choice to solve large-scale optimal transport (OT) problems. In this context, it involves an inverse temperature parameter $\beta$ that determines the speed-accuracy trade-off. To improve this trade-off, practitioners often use a variant of this algorithm, Annealed Sinkhorn, that uses an nondecreasing sequence $(\beta_t)_{t\in \mathbb{N}}$ where $t$ is the iteration count. However, besides for the schedule $\beta_t=\Theta(\log t)$ which is impractically slow, it is not known whether this variant is guaranteed to actually solve OT. Our first contribution answers this question: we show that a concave annealing schedule asymptotically solves OT if and only if $\beta_t\to+\infty$ and $\beta_t-\beta_{t-1}\to 0$. The proof is based on an equivalence with Online Mirror Descent and further suggests that the iterates of Annealed Sinkhorn follow the solutions of a sequence of relaxed, entropic OT problems, the regularization path. An analysis of this path reveals that, in addition to the well-known "entropic" error in $\Theta(\beta^{-1}_t)$, the annealing procedure induces a "relaxation" error in $\Theta(\beta_{t}-\beta_{t-1})$. The best error trade-off is achieved with the schedule $\beta_t = \Theta(\sqrt{t})$ which, albeit slow, is a universal limitation of this method. Going beyond this limitation, we propose a simple modification of Annealed Sinkhorn that reduces the relaxation error, and therefore enables faster annealing schedules. In toy experiments, we observe the effectiveness of our Debiased Annealed Sinkhorn's algorithm: a single run of this algorithm spans the whole speed-accuracy Pareto front of the standard Sinkhorn's algorithm.

Machine Learning,Optimization and Control
Quantitative contraction rates for Sinkhorn algorithm: beyond bounded costs and compact marginals

Giovanni Conforti,Alain Durmus,Giacomo Greco

2024-06-21

Abstract:We show non-asymptotic exponential convergence of Sinkhorn iterates to the Schrödinger potentials, solutions of the quadratic Entropic Optimal Transport problem on $\mathbb{R}^d$. Our results hold under mild assumptions on the marginal inputs: in particular, we only assume that they admit an asymptotically positive log-concavity profile, covering as special cases log-concave distributions and bounded smooth perturbations of quadratic potentials. Up to the authors' knowledge, these are the first results which establish exponential convergence of Sinkhorn's algorithm in a general setting without assuming bounded cost functions or compactly supported marginals.

Probability,Optimization and Control
Sharper Exponential Convergence Rates for Sinkhorn's Algorithm in Continuous Settings

Lénaïc Chizat,Alex Delalande,Tomas Vaškevičius

2024-07-01

Abstract:We study the convergence rate of Sinkhorn's algorithm for solving entropy-regularized optimal transport problems when at least one of the probability measures, $\mu$, admits a density over $\mathbb{R}^d$. For a semi-concave cost function bounded by $c_{\infty}$ and a regularization parameter $\lambda > 0$, we obtain exponential convergence guarantees on the dual sub-optimality gap with contraction rate polynomial in $\lambda/c_{\infty}$. This represents an exponential improvement over the known contraction rate $1 - \Theta(\exp(-c_{\infty}/\lambda))$ achievable via Hilbert's projective metric. Specifically, we prove a contraction rate value of $1-\Theta(\lambda^2/c_\infty^2)$ when $\mu$ has a bounded log-density. In some cases, such as when $\mu$ is log-concave and the cost function is $c(x,y)=-\langle x, y \rangle$, this rate improves to $1-\Theta(\lambda/c_\infty)$. The latter rate matches the one that we derive for the transport between isotropic Gaussian measures, indicating tightness in the dependency in $\lambda/c_\infty$. Our results are fully non-asymptotic and explicit in all the parameters of the problem.

Optimization and Control
Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

Benjamin Grimmer,Kevin Shu,Alex L. Wang

2024-04-12

Abstract:This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the recent silver stepsizes of Altschuler and Parrilo. For one of our stepsize sequences, we prove a $O\left(N^{- 1.2716\dots}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for squared-gradient-norm decrease. This first result improves on the recent result of Altschuler and Parrilo by a constant factor, while the second results improve on the exponent of the prior best squared-gradient-norm convergence guarantee of $O\left(N^{-1}\right)$.

Optimization and Control
Nested Sinkhorn Divergence To Compute The Nested Distance

Alois Pichler,Michael Weinhardt

DOI: https://doi.org/10.48550/arXiv.2102.05413

2021-02-10

Abstract:The nested distance builds on the Wasserstein distance to quantify the difference of stochastic processes, including also the information modelled by filtrations. The Sinkhorn divergence is a relaxation of the Wasserstein distance, which can be computed considerably faster. For this reason we employ the Sinkhorn divergence and take advantage of the related (fixed point) iteration algorithm. Furthermore, we investigate the transition of the entropy throughout the stages of the stochastic process and provide an entropy-regularized nested distance formulation, including a characterization of its dual. Numerical experiments affirm the computational advantage and supremacy.

Optimization and Control
Neural Sinkhorn Gradient Flow

Huminhao Zhu,Fangyikang Wang,Chao Zhang,Hanbin Zhao,Hui Qian

2024-01-25

Abstract:Wasserstein Gradient Flows (WGF) with respect to specific functionals have been widely used in the machine learning literature. Recently, neural networks have been adopted to approximate certain intractable parts of the underlying Wasserstein gradient flow and result in efficient inference procedures. In this paper, we introduce the Neural Sinkhorn Gradient Flow (NSGF) model, which parametrizes the time-varying velocity field of the Wasserstein gradient flow w.r.t. the Sinkhorn divergence to the target distribution starting a given source distribution. We utilize the velocity field matching training scheme in NSGF, which only requires samples from the source and target distribution to compute an empirical velocity field approximation. Our theoretical analyses show that as the sample size increases to infinity, the mean-field limit of the empirical approximation converges to the true underlying velocity field. To further enhance model efficiency on high-dimensional tasks, a two-phase NSGF++ model is devised, which first follows the Sinkhorn flow to approach the image manifold quickly ($\le 5$ NFEs) and then refines the samples along a simple straight flow. Numerical experiments with synthetic and real-world benchmark datasets support our theoretical results and demonstrate the effectiveness of the proposed methods.

Machine Learning

A Gradient Descent Perspective on Sinkhorn

Sinkhorn Barycenter via Functional Gradient Descent

Sinkhorn Flow: A Continuous-Time Framework for Understanding and Generalizing the Sinkhorn Algorithm

Non-asymptotic convergence bounds for Sinkhorn iterates and their gradients: a coupling approach

Sinkhorn Natural Gradient for Generative Models

On the geometry and dynamical formulation of the Sinkhorn algorithm for optimal transport

Accelerating Sinkhorn Algorithm with Sparse Newton Iterations

Re-basin via implicit Sinkhorn differentiation

Mirror and Preconditioned Gradient Descent in Wasserstein Space

The Information Geometry of Mirror Descent

Gradient is All You Need?

Rethinking Initialization of the Sinkhorn Algorithm

A Mirror Descent Perspective of Smoothed Sign Descent

A semiconcavity approach to stability of entropic plans and exponential convergence of Sinkhorn's algorithm

Linear Time Sinkhorn Divergences using Positive Features

Annealed Sinkhorn for Optimal Transport: convergence, regularization path and debiasing

Quantitative contraction rates for Sinkhorn algorithm: beyond bounded costs and compact marginals

Sharper Exponential Convergence Rates for Sinkhorn's Algorithm in Continuous Settings

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

Nested Sinkhorn Divergence To Compute The Nested Distance

Neural Sinkhorn Gradient Flow