Fisher-Riemann geometry for nonparametric probability densities

Hugo Aimar,Aníbal Chicco Ruiz,Ivana Gómez

2024-02-17

Abstract:In this article we aim to obtain the Fisher Riemann geodesics for nonparametric families of probability densities as a weak limit of the parametric case with increasing number of parameters.

Classical Analysis and ODEs,Information Theory,Differential Geometry

What problem does this paper attempt to address?

The nonparametric Fisher geometry and the chi-square process density prior

Andrew Holbrook,Shiwei Lan,Jeffrey Streets,Babak Shahbaba

DOI: https://doi.org/10.48550/arXiv.1707.03117

2017-11-15

Abstract:It is well known that the Fisher information induces a Riemannian geometry on parametric families of probability density functions. Following recent work, we consider the nonparametric generalization of the Fisher geometry. The resulting nonparametric Fisher geometry is shown to be equivalent to a familiar, albeit infinite-dimensional, geometric object---the sphere. By shifting focus away from density functions and toward \emph{square-root} density functions, one may calculate theoretical quantities of interest with ease. More importantly, the sphere of square-root densities is much more computationally tractable. This insight leads to a novel Bayesian nonparametric density estimation model. We construct the $\chi^2$-process density prior by modeling the square-root density with a restricted Gaussian process prior. Inference over square-root densities is fast, and the model retains the flexibility characteristic of Bayesian nonparametric models. Finally, we formalize the relationship between spherical HMC in the infinite-dimensional limit and standard Riemannian HMC.

Methodology
Riemannian Laplace Approximation with the Fisher Metric

Hanlin Yu,Marcelo Hartmann,Bernardo Williams,Mark Girolami,Arto Klami

2024-05-07

Abstract:Laplace's method approximates a target density with a Gaussian distribution at its mode. It is computationally efficient and asymptotically exact for Bayesian inference due to the Bernstein-von Mises theorem, but for complex targets and finite-data posteriors it is often too crude an approximation. A recent generalization of the Laplace Approximation transforms the Gaussian approximation according to a chosen Riemannian geometry providing a richer approximation family, while still retaining computational efficiency. However, as shown here, its properties depend heavily on the chosen metric, indeed the metric adopted in previous work results in approximations that are overly narrow as well as being biased even at the limit of infinite data. We correct this shortcoming by developing the approximation family further, deriving two alternative variants that are exact at the limit of infinite data, extending the theoretical analysis of the method, and demonstrating practical improvements in a range of experiments.

Machine Learning,Methodology
The Fisher Geometry and Geodesics of the Multivariate Normals, without Differential Geometry

Brodie A. J. Lawson,Kevin Burrage,Kerrie Mengersen,Rodrigo Weber dos Santos

2023-06-02

Abstract:Choosing the Fisher information as the metric tensor for a Riemannian manifold provides a powerful yet fundamental way to understand statistical distribution families. Distances along this manifold become a compelling measure of statistical distance, and paths of shorter distance improve sampling techniques that leverage a sequence of distributions in their operation. Unfortunately, even for a distribution as generally tractable as the multivariate normal distribution, this information geometry proves unwieldy enough that closed-form solutions for shortest-distance paths or their lengths remain unavailable outside of limited special cases. In this review we present for general statisticians the most practical aspects of the Fisher geometry for this fundamental distribution family. Rather than a differential geometric treatment, we use an intuitive understanding of the covariance-induced curvature of this manifold to unify the special cases with known closed-form solution and review approximate solutions for the general case. We also use the multivariate normal information geometry to better understand the paths or distances commonly used in statistics (annealing, Wasserstein). Given the unavailability of a general solution, we also discuss the methods used for numerically obtaining geodesics in the space of multivariate normals, identifying remaining challenges and suggesting methodological improvements.

Statistics Theory,Computation
The Fisher-Rao geometry of CES distributions

Florent Bouchard,Arnaud Breloy,Antoine Collas,Alexandre Renaux,Guillaume Ginolhac

2023-10-02

Abstract:When dealing with a parametric statistical model, a Riemannian manifold can naturally appear by endowing the parameter space with the Fisher information metric. The geometry induced on the parameters by this metric is then referred to as the Fisher-Rao information geometry. Interestingly, this yields a point of view that allows for leveragingmany tools from differential geometry. After a brief introduction about these concepts, we will present some practical uses of these geometric tools in the framework of elliptical distributions. This second part of the exposition is divided into three main axes: Riemannian optimization for covariance matrix estimation, Intrinsic Cramér-Rao bounds, and classification using Riemannian distances.

Machine Learning,Applications
Fisher-Rao geometry of equivalent Gaussian measures on infinite-dimensional Hilbert spaces

Minh Ha Quang

2023-10-16

Abstract:This work presents an explicit description of the Fisher-Rao Riemannian metric on the Hilbert manifold of equivalent centered Gaussian measures on an infinite-dimensional Hilbert space. We show that the corresponding quantities from the finite-dimensional setting of Gaussian densities on Euclidean space, including the Riemannian metric, Levi-Civita connection, curvature, geodesic curve, and Riemannian distance, when properly formulated, directly generalize to this setting. Furthermore, we discuss the connection with the Riemannian geometry of positive definite unitized Hilbert-Schmidt operators on Hilbert space, which can be viewed as a regularized version of the current setting.

Probability,Differential Geometry
Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalities

José A. Carrillo,Yifan Chen,Daniel Zhengyu Huang,Jiaoyang Huang,Dongyi Wei

2024-07-22

Abstract:The dynamics of probability density functions has been extensively studied in science and engineering to understand physical phenomena and facilitate algorithmic design. Of particular interest are dynamics that can be formulated as gradient flows of energy functionals under the Wasserstein metric. The development of functional inequalities, such as the log-Sobolev inequality, plays a pivotal role in analyzing the convergence of these dynamics. The goal of this paper is to parallel the success of techniques using functional inequalities, for dynamics that are gradient flows under the Fisher-Rao metric, with various $f$-divergences as energy functionals. Such dynamics take the form of a nonlocal differential equation, for which existing analysis critically relies on using the explicit solution formula in special cases. We provide a comprehensive study on functional inequalities and the relevant geodesic convexity for Fisher-Rao gradient flows under minimal assumptions. A notable feature of the obtained functional inequalities is that they do not depend on the log-concavity or log-Sobolev constants of the target distribution. Consequently, the convergence rate of the dynamics (assuming well-posed) is uniform across general target distributions, making them potentially desirable dynamics for posterior sampling applications in Bayesian inference.

Analysis of PDEs
Fisher-Rao geometry and Jeffreys prior for Pareto distribution

Mingming Li,Huafei Sun,Linyu Peng

DOI: https://doi.org/10.1080/03610926.2020.1771593

2022-01-01

Abstract:In this paper, we investigate the Fisher-Rao geometry of the two-parameter family of Pareto distribution. We prove that its geometrical structure is isometric to the Poincare upper half-plane model, and then study the corresponding geometrical features by presenting explicit expressions for connection, curvature and geodesics. It is then applied to Bayesian inference by considering the Jeffreys prior determined by the volume form. In addition, the posterior distribution from the prior is computed, providing a systematic method to the Bayesian inference for Pareto distribution.
Any Kähler metric is a Fisher information metric

Emmanuel Gnandi

2024-05-29

Abstract:The Fisher information metric or the Fisher-Rao metric corresponds to a natural Riemannian metric defined on a parameterized family of probability density functions. As in the case of Riemannian geometry, we can define a distance in terms of the Fisher information metric, called the Fisher-Rao distance. The Fisher information metric has a wide range of applications in estimation and information theories. Indeed, it provides the most informative Cramer-Rao bound for an unbiased estimator. The Goldberg conjecture is a well-known unsolved problem which states that any compact Einstein almost Kähler manifold is necessarily a Kähler-Einstein. Note that, there is also a known odd-dimensional analog of the Goldberg conjecture in the literature. The main objective of this paper is to establish a new characterization of coKähler manifolds and Kähler manifolds; our characterization is statistical in nature. Finally, we corroborate that every, Kähler and co-Kähler manifolds, can be viewed as being a parametric family of probability density functions, whereas Kähler and coKähler metrics can be regarded as Fisher information metrics. In particular, we prove that, when the Kähler metric is real analytic, it is always locally the Fisher information of an exponential family. We also tackle the link between Kähler potential and Kullback-Leibler divergence.

Differential Geometry
The Riemannian geometry of Sinkhorn divergences

Hugo Lavenant,Jonas Luckhardt,Gilles Mordant,Bernhard Schmitzer,Luca Tamanini

2024-05-08

Abstract:We propose a new metric between probability measures on a compact metric space that mirrors the Riemannian manifold-like structure of quadratic optimal transport but includes entropic regularization. Its metric tensor is given by the Hessian of the Sinkhorn divergence, a debiased variant of entropic optimal transport. We precisely identify the tangent space it induces, which turns out to be related to a Reproducing Kernel Hilbert Space (RKHS). As usual in Riemannian geometry, the distance is built by looking for shortest paths. We prove that our distance is geodesic, metrizes the weak-star topology, and is equivalent to a RKHS norm. Still it retains the geometric flavor of optimal transport: as a paradigmatic example, translations are geodesics for the quadratic cost on $\mathbb{R}^d$. We also show two negative results on the Sinkhorn divergence that may be of independent interest: that it is not jointly convex, and that its square root is not a distance because it fails to satisfy the triangle inequality.

Optimization and Control,Metric Geometry
Wasserstein geometry and Ricci curvature bounds for Poisson spaces

Lorenzo Dello Schiavo,Ronan Herry,Kohei Suzuki

2023-03-01

Abstract:Let $\varUpsilon$ be the configuration space over a complete and separable metric base space, endowed with the Poisson measure $\pi$. We study the geometry of $\varUpsilon$ from the point of view of optimal transport and Ricci-lower bounds. To do so, we define a formal Riemannian structure on $\mathscr{P}_{1}(\varUpsilon)$, the space of probability measures over $\varUpsilon$ with finite first moment, and we construct an extended distance $\mathcal{W}$ on $\mathscr{P}_{1}(\varUpsilon)$. The distance $\mathcal{W}$ corresponds, in our setting, to the Benamou--Brenier variational formulation of the Wasserstein distance. Our main technical tool is a non-local continuity equation defined via the difference operator on the Poisson space. We show that the closure of the domain of the relative entropy is a complete geodesic space, when endowed with $\mathcal{W}$. We establish non-local infinite-dimensional analogues of results regarding the geometry of the Wasserstein space over a metric measure space with synthetic Ricci curvature bounded below. In particular, we obtain that: (a) the Ornstein--Uhlenbeck semi-group is the gradient flow of the relative entropy; (b) the Poisson space has a Ricci curvature, in the entropic sense, bounded below by $1$; (c) the distance $\mathcal{W}$ satisfies an HWI inequality.

Probability,Functional Analysis
Approximation and bounding techniques for the Fisher-Rao distances between parametric statistical models

Frank Nielsen

2024-05-22

Abstract:The Fisher-Rao distance between two probability distributions of a statistical model is defined as the Riemannian geodesic distance induced by the Fisher information metric. In order to calculate the Fisher-Rao distance in closed-form, we need (1) to elicit a formula for the Fisher-Rao geodesics, and (2) to integrate the Fisher length element along those geodesics. We consider several numerically robust approximation and bounding techniques for the Fisher-Rao distances: First, we report generic upper bounds on Fisher-Rao distances based on closed-form 1D Fisher-Rao distances of submodels. Second, we describe several generic approximation schemes depending on whether the Fisher-Rao geodesics or pregeodesics are available in closed-form or not. In particular, we obtain a generic method to guarantee an arbitrarily small additive error on the approximation provided that Fisher-Rao pregeodesics and tight lower and upper bounds are available. Third, we consider the case of Fisher metrics being Hessian metrics, and report generic tight upper bounds on the Fisher-Rao distances using techniques of information geometry. Uniparametric and biparametric statistical models always have Fisher Hessian metrics, and in general a simple test allows to check whether the Fisher information matrix yields a Hessian metric or not. Fourth, we consider elliptical distribution families and show how to apply the above techniques to these models. We also propose two new distances based either on the Fisher-Rao lengths of curves serving as proxies of Fisher-Rao geodesics, or based on the Birkhoff/Hilbert projective cone distance. Last, we consider an alternative group-theoretic approach for statistical transformation models based on the notion of maximal invariant which yields insights on the structures of the Fisher-Rao distance formula which may be used fruitfully in applications.

Information Theory,Computer Vision and Pattern Recognition,Machine Learning
Cubic Hermite interpolators on the space of probability measures

Ines Adouani,Chafik Samir,Tien Tam Tran

DOI: https://doi.org/10.1002/mma.10240

2024-06-17

Mathematical Methods in the Applied Sciences

Abstract:In this paper, we introduce a novel two‐step modeling method to generalize cubic Hermite interpolators on the space of probability measures P+(I) . First, we develop new approaches to capture the Riemannian geometric structure of P+(I) when equipped with Fisher–Rao metric. Furthermore, we develop and detail all numerical tools on P+(I) , namely, Levi–Civita connection, minimal geodesics, parallel transport, exponential map, and logarithm map. Then, we demonstrate that preliminary analysis results yield significant benefits in constructing an optimal cubic Hermite spline on P+(I) as a nonlinear Riemannian manifold, precisely where conventional numerical methods fail.

mathematics, applied
Nonparametric Information Geometry: From Divergence Function to Referential-Representational Biduality on Statistical Manifolds

Jun Zhang

DOI: https://doi.org/10.3390/e15125384

IF: 2.738

2013-12-04

Entropy

Abstract:Divergence functions are the non-symmetric “distance” on the manifold, Μθ, of parametric probability density functions over a measure space, (Χ,μ). Classical information geometry prescribes, on Μθ: (i) a Riemannian metric given by the Fisher information; (ii) a pair of dual connections (giving rise to the family of α-connections) that preserve the metric under parallel transport by their joint actions; and (iii) a family of divergence functions ( α-divergence) defined on Μθ x Μθ, which induce the metric and the dual connections. Here, we construct an extension of this differential geometric structure from Μθ (that of parametric probability density functions) to the manifold, Μ, of non-parametric functions on X, removing the positivity and normalization constraints. The generalized Fisher information and α-connections on M are induced by an α-parameterized family of divergence functions, reflecting the fundamental convex inequality associated with any smooth and strictly convex function. The infinite-dimensional manifold, M, has zero curvature for all these α-connections; hence, the generally non-zero curvature of M can be interpreted as arising from an embedding of Μθ into Μ. Furthermore, when a parametric model (after a monotonic scaling) forms an affine submanifold, its natural and expectation parameters form biorthogonal coordinates, and such a submanifold is dually flat for α = ± 1, generalizing the results of Amari’s α-embedding. The present analysis illuminates two different types of duality in information geometry, one concerning the referential status of a point (measurable function) expressed in the divergence function (“referential duality”) and the other concerning its representation under an arbitrary monotone scaling (“representational duality”).

Computer Science
Some Inequalities on Riemannian Manifolds Linking Entropy, Fisher Information, Stein Discrepancy and Wasserstein Distance

Li-Juan Cheng,Anton Thalmaier,Feng-Yu Wang

DOI: https://doi.org/10.1016/j.jfa.2023.109997

IF: 1.891

2023-01-01

Journal of Functional Analysis

Abstract:For a complete connected Riemannian manifold M let V is an element of C2(M) be such that mu(dx) = e-V(x)vol(dx) is a probabil-ity measure on M. Taking mu as reference measure, we derive inequalities for probability measures on M linking relative entropy, Fisher information, Stein discrepancy and Wasser-stein distance. These inequalities strengthen in particular the famous log-Sobolev and transportation-cost inequality and extend the so-called Entropy/Stein-discrepancy/Information (HSI) inequality established by Ledoux, Nourdin and Peccati (2015) for the standard Gaussian measure on Euclidean space to the setting of Riemannian manifolds.(c) 2023 Elsevier Inc. All rights reserved.
$L^\infty$-optimal transport of anisotropic log-concave measures and exponential convergence in Fisher's infinitesimal model

Ksenia A. Khudiakova,Jan Maas,Francesco Pedrotti

2024-02-07

Abstract:We prove upper bounds on the $L^\infty$-Wasserstein distance from optimal transport between strongly log-concave probability densities and log-Lipschitz perturbations. In the simplest setting, such a bound amounts to a transport-information inequality involving the $L^\infty$-Wasserstein metric and the relative $L^\infty$-Fisher information. We show that this inequality can be sharpened significantly in situations where the involved densities are anisotropic. Our proof is based on probabilistic techniques using Langevin dynamics. As an application of these results, we obtain sharp exponential rates of convergence in Fisher's infinitesimal model from quantitative genetics, generalising recent results by Calvez, Poyato, and Santambrogio in dimension 1 to arbitrary dimensions.

Probability
Fisher-Rao distance on the covariance cone

Joseph Wells,Mary Cook,Karleigh Pine,Benjamin D. Robinson

DOI: https://doi.org/10.48550/arXiv.2010.15861

2020-10-30

Abstract:The Fisher-Rao geodesic distance on the statistical manifold consisting of zero-mean p-dimensional multivariate Gaussians appears without proof in several places (such as Steven Smith's "Covariance, Subspace, and Intrinsic Cramer-Rao Bounds"). In this paper, we give a proof using basic Riemannian geometry.

Statistics Theory,Differential Geometry
The Fisher metric as a metric on the cotangent bundle

Hiroshi Nagaoka

DOI: https://doi.org/10.48550/arXiv.2310.13237

2023-10-20

Abstract:The Fisher metric on a manifold of probability distributions is usually treated as a metric on the tangent bundle. In this paper, we focus on the metric on the cotangent bundle induced from the Fisher metric with calling it the Fisher co-metric. We show that the Fisher co-metric can be defined directly without going through the Fisher metric by establishing a natural correspondence between cotangent vectors and random variables. This definition clarifies a close relation between the Fisher co-metric and the variance/covariance of random variables, whereby the Cramér-Rao inequality is trivialized. We also discuss the monotonicity and the invariance of the Fisher co-metric with respect to Markov maps, and present a theorem characterizing the co-metric by the invariance, which can be regarded as a cotangent version of Čencov's characterization theorem for the Fisher metric. The obtained theorem can also viewed as giving a characterization of the variance/covariance.

Information Theory,Differential Geometry,Probability,Statistics Theory
Transversely Hessian foliations and information geometry

Michel Nguiffo Boyom,Robert A. Wolak

DOI: https://doi.org/10.1142/s0129167x16500920

2016-10-01

International Journal of Mathematics

Abstract:A family of probability distributions parametrized by an open domain [Formula: see text] in [Formula: see text] defines the Fisher information matrix on this domain which is positive semi-definite. In information geometry, the standard assumption has been that the Fisher information matrix tensor is positive definite defining in this way a Riemannian metric on [Formula: see text]. It seems to be quite a strong condition. In general, not much can be said about the Fisher information matrix tensor. To develop a more general theory, we weaken the assumption and replace “positive definite” by the existence of a suitable torsion-free connection. It permits us to define naturally a foliation with a transversely Hessian structure. We develop the theory of transversely Hessian foliations along the lines of the classical theory.

mathematics
A numerical approximation method for the Fisher-Rao distance between multivariate normal distributions

Frank Nielsen

DOI: https://doi.org/10.48550/arXiv.2302.08175

2023-02-16

Information Theory

Abstract:We present a simple method to approximate Rao's distance between multivariate normal distributions based on discretizing curves joining normal distributions and approximating Rao's distances between successive nearby normal distributions on the curves by the square root of Jeffreys divergence, the symmetrized Kullback-Leibler divergence. We consider experimentally the linear interpolation curves in the ordinary, natural and expectation parameterizations of the normal distributions, and compare these curves with a curve derived from the Calvo and Oller's isometric embedding of the Fisher-Rao $d$-variate normal manifold into the cone of $(d+1)\times (d+1)$ symmetric positive-definite matrices [Journal of multivariate analysis 35.2 (1990): 223-242]. We report on our experiments and assess the quality of our approximation technique by comparing the numerical approximations with both lower and upper bounds. Finally, we present several information-geometric properties of the Calvo and Oller's isometric embedding.
Some informational inequalities involving generalized trigonometric functions and a new class of generalized moments

David Puertas-Centeno,Steeve Zozor

2024-11-01

Abstract:In this work, we define a family of probability densities involving the generalized trigonometric functions defined by Drábek and Manásevich, we name Generalized Trigonometric Densities. We show their relationship with the generalized stretched Gaussians and other types of laws such as logistic, hyperbolic secant, and raised cosine probability densities. We prove that, for a fixed generalized Fisher information, this family of densities is of minimal Rényi entropy. Moreover, we introduce generalized moments via the mean of the power of a deformed cumulative distribution. The latter is defined as a cumulative of the power of the probability density function, this second parameter tuning the tail weight of the deformed cumulative distribution. These generalized moments coincide with the usual moments of a deformed probability distribution with a regularized tail. We show that, for any bounded probability density, there exists a critical value for this second parameter below which the whole subfamily of generalized moments is finite for any positive value of the first parameter (power of the moment). In addition, we show that such generalized moments satisfy remarkable properties like order relation w.r.t. the first parameter, or adequate scaling behavior. Finally, we highlight that, if we constrain such a generalized moment, both the Rényi entropy and generalized Fisher information achieve respectively their maximum and minimum for the generalized trigonometric densities.

Mathematical Physics

Fisher-Riemann geometry for nonparametric probability densities

The nonparametric Fisher geometry and the chi-square process density prior

Riemannian Laplace Approximation with the Fisher Metric

The Fisher Geometry and Geodesics of the Multivariate Normals, without Differential Geometry

The Fisher-Rao geometry of CES distributions

Fisher-Rao geometry of equivalent Gaussian measures on infinite-dimensional Hilbert spaces

Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalities

Fisher-Rao geometry and Jeffreys prior for Pareto distribution

Any Kähler metric is a Fisher information metric

The Riemannian geometry of Sinkhorn divergences

Wasserstein geometry and Ricci curvature bounds for Poisson spaces

Approximation and bounding techniques for the Fisher-Rao distances between parametric statistical models

Cubic Hermite interpolators on the space of probability measures

Nonparametric Information Geometry: From Divergence Function to Referential-Representational Biduality on Statistical Manifolds

Some Inequalities on Riemannian Manifolds Linking Entropy, Fisher Information, Stein Discrepancy and Wasserstein Distance

$L^\infty$-optimal transport of anisotropic log-concave measures and exponential convergence in Fisher's infinitesimal model

Fisher-Rao distance on the covariance cone

The Fisher metric as a metric on the cotangent bundle

Transversely Hessian foliations and information geometry

A numerical approximation method for the Fisher-Rao distance between multivariate normal distributions

Some informational inequalities involving generalized trigonometric functions and a new class of generalized moments