Abstract:Applications such as unbalanced and fully shuffled regression can be approached by optimizing regularized optimal transport (OT) distances, such as the entropic OT and Sinkhorn distances. A common approach for this optimization is to use a first-order optimizer, which requires the gradient of the OT distance. For faster convergence, one might also resort to a second-order optimizer, which additionally requires the Hessian. The computations of these derivatives are crucial for efficient and accurate optimization. However, they present significant challenges in terms of memory consumption and numerical instability, especially for large datasets and small regularization strengths. We circumvent these issues by analytically computing the gradients for OT distances and the Hessian for the entropic OT distance, which was not previously used due to intricate tensor-wise calculations and the complex dependency on parameters within the bi-level loss function. Through analytical derivation and spectral analysis, we identify and resolve the numerical instability caused by the singularity and ill-posedness of a key linear system. Consequently, we achieve scalable and stable computation of the Hessian, enabling the implementation of the stochastic gradient descent (SGD)-Newton methods. Tests on shuffled regression examples demonstrate that the second stage of the SGD-Newton method converges orders of magnitude faster than the gradient descent-only method while achieving significantly more accurate parameter estimations.

Numerical Differentiation for the Second Order Derivatives of Functions of Two Variables

First and Second Order Numerical Differentiation with Tikhonov Regularization

Numerical Differentiation from A Viewpoint of Regularization Theory

Numerical Simulation for the System of Ordinary Differential Equations

Numerical Differentiation by a Tikhonov Regularization Method Based on the Discrete Cosine Transform

Numerical Differentiation and Its Applications

A Numerical Differentiation Method and Its Application to Reconstruction of Discontinuity

Legendre Polynomials As A Recommended Basis for Numerical Differentiation in the Presence of Stochastic White Noise

On regularized Radon-Nikodym differentiation

Robust First and Second-Order Differentiation for Regularized Optimal Transport

A Second-Order Finite-Difference Method for Derivative-Free Optimization

A note about high-order semi-implicit differentiation: application to a numerical integration scheme with Taylor-based compensated error

Second-order subdifferential calculus with applications to tilt stability in optimization

Using second-order information in gradient sampling methods for nonsmooth optimization

A set of compact finite-difference approximations to first and second derivatives, related to the extended Numerov method of Chawla on nonuniform grids

Numerical Analysis of a High-Order Scheme for Nonlinear Fractional Differential Equations with Uniform Accuracy

Formulas for Numerical Differentiation on a Uniform Mesh in the Presence of a Boundary Layer

An Optimal Method for High-Order Mixed Derivatives of Bivariate Functions

Numerical differentiation by the polynomial-exponential basis

A second order numerical method for a Volterra integro-differential equation with a weakly singular kernel

Numerical analysis of a nonsmooth quasilinear elliptic control problem: I. Explicit second-order optimality conditions