Optimization via Information Geometry

Luigi Malagò,Giovanni Pistone

DOI: https://doi.org/10.48550/arXiv.1403.3762

2014-03-15

Statistics Theory

Abstract:Information Geometry has been used to inspire efficient algorithms for stochastic optimization, both in the combinatorial and the continuous case. We give an overview of the authors' research program and some specific contributions to the underlying theory.

What problem does this paper attempt to address?

Information Geometry for the Working Information Theorist

Kumar Vijay Mishra,M. Ashok Kumar,Ting-Kam Leonard Wong

2023-10-06

Abstract:Information geometry is a study of statistical manifolds, that is, spaces of probability distributions from a geometric perspective. Its classical information-theoretic applications relate to statistical concepts such as Fisher information, sufficient statistics, and efficient estimators. Today, information geometry has emerged as an interdisciplinary field that finds applications in diverse areas such as radar sensing, array signal processing, quantum physics, deep learning, and optimal transport. This article presents an overview of essential information geometry to initiate an information theorist, who may be unfamiliar with this exciting area of research. We explain the concepts of divergences on statistical manifolds, generalized notions of distances, orthogonality, and geodesics, thereby paving the way for concrete applications and novel theoretical investigations. We also highlight some recent information-geometric developments, which are of interest to the broader information theory community.

Information Theory,Machine Learning,Signal Processing,Differential Geometry
Progress in Theory and Applications of Information Geometry

黎湘,程永强,王宏强,秦玉亮

DOI: https://doi.org/10.1360/112012-424

2013-01-01

Scientia Sinica Informationis

Abstract:Information geometry is the fundamental and cutting-edge discipline that explores statistical problems on Riemannian manifolds of probability distributions using the methods of differential geometry. It has been identified as the second generation of modern information theory pioneered by Shannon, and exhibits great potential in developing the field of information science and systems theory. This paper begins defining the scientific content of information geometry from the intrinsic geometrical structures of parameterized families of probability distributions as well as geometric properties of information; it points out theoretical advantages and methodological innovations of information geometry compared with classical statistics and information theory. Next, the paper briefly introduces connections between information geometry and differential geometry, and elaborates the history of information geometry as well as its applications to various areas such as neural networks, statistical inference, communications and coding, systems and control theory, physics and medical imaging, among others. In particular, the applications of information geometry to signal processing, including the latest results of geometric methods of signal detection, nonlinear parameter estimation, and filtering, are comprehensively introduced. The basic ideas and methods of information geometry are also summarized. Finally, by viewing prospects for the development of information geometry, several open problems of information geometry in applications to signal processing are proposed.
Information Geometry of the Probability Simplex: A Short Course

Giovanni Pistone

DOI: https://doi.org/10.33581/1561-4085-2020-23-2-221-242

2020-07-09

Nonlinear Phenomena in Complex Systems

Abstract:This set of notes is intended for a short course aiming to provide an (almost) self-contained and (almost) elementary introduction to the topic of Information Geometry (IG) of the probability simplex. Such a course can be considered an introduction to the original monograph by Amari and Nagaoka [1], and to the recent monographs by Amari [2] and by Ay, Jost, Lê, and Schwachhöfer [3]. The focus is on a non-parametric approach, that is, I consider the geometry of the full probability simplex and compare the IG formalism with what is classically done in Statistical Physics.
Geometry, Computation, and Optimality in Stochastic Optimization

Chen Cheng,Daniel Levy,John C. Duchi

2024-10-12

Abstract:We study computational and statistical consequences of problem geometry in stochastic and online optimization. By focusing on constraint set and gradient geometry, we characterize the problem families for which stochastic- and adaptive-gradient methods are (minimax) optimal and, conversely, when nonlinear updates -- such as those mirror descent employs -- are necessary for optimal convergence. When the constraint set is quadratically convex, diagonally pre-conditioned stochastic gradient methods are minimax optimal. We provide quantitative converses showing that the ``distance'' of the underlying constraints from quadratic convexity determines the sub-optimality of subgradient methods. These results apply, for example, to any $\ell_p$-ball for $p < 2$, and the computation/accuracy tradeoffs they demonstrate exhibit a striking analogy to those in Gaussian sequence models.

Optimization and Control,Information Theory,Machine Learning
An Invariant Information Geometric Method for High-Dimensional Online Optimization

Zhengfei Zhang,Yunyue Wei,Yanan Sui

2024-01-03

Abstract:Sample efficiency is crucial in optimization, particularly in black-box scenarios characterized by expensive evaluations and zeroth-order feedback. When computing resources are plentiful, Bayesian optimization is often favored over evolution strategies. In this paper, we introduce a full invariance oriented evolution strategies algorithm, derived from its corresponding framework, that effectively rivals the leading Bayesian optimization method in tasks with dimensions at the upper limit of Bayesian capability. Specifically, we first build the framework InvIGO that fully incorporates historical information while retaining the full invariant and computational complexity. We then exemplify InvIGO on multi-dimensional Gaussian, which gives an invariant and scalable optimizer SynCMA . The theoretical behavior and advantages of our algorithm over other Gaussian-based evolution strategies are further analyzed. Finally, We benchmark SynCMA against leading algorithms in Bayesian optimization and evolution strategies on various high dimension tasks, in cluding Mujoco locomotion tasks, rover planning task and synthetic functions. In all scenarios, SynCMA demonstrates great competence, if not dominance, over other algorithms in sample efficiency, showing the underdeveloped potential of property oriented evolution strategies.

Machine Learning,Neural and Evolutionary Computing
Dually affine Information Geometry modeled on a Banach space

Goffredo Chirco,Giovanni Pistone

2024-05-13

Abstract:In this chapter, we study Information Geometry from a particular non-parametric or functional point of view. The basic model is a probabilities subset usually specified by regularity conditions. For example, probability measures mutually absolutely continuous or probability densities with a given degree of smoothness. We construct a manifold structure by giving an atlas of charts as mappings from probabilities to a Banach space. The charts we use are quite peculiar in that we consider only instances where the transition mappings are affine. We chose a particular expression of the tangent and cotangent bundles in this affine setting.

Statistics Theory
Information Geometry of Dynamics on Graphs and Hypergraphs

Tetsuya J. Kobayashi,Dimitri Loutchko,Atsushi Kamimura,Shuhei A. Horiguchi,Yuki Sughiyama

2023-08-05

Abstract:We introduce a new information-geometric structure associated with the dynamics on discrete objects such as graphs and hypergraphs. The presented setup consists of two dually flat structures built on the vertex and edge spaces, respectively. The former is the conventional duality between density and potential, e.g., the probability density and its logarithmic form induced by a convex thermodynamic function. The latter is the duality between flux and force induced by a convex and symmetric dissipation function, which drives the dynamics of the density. These two are connected topologically by the homological algebraic relation induced by the underlying discrete objects. The generalized gradient flow in this doubly dual flat structure is an extension of the gradient flows on Riemannian manifolds, which include Markov jump processes and nonlinear chemical reaction dynamics as well as the natural gradient and mirror descent. The information-geometric projections on this doubly dual flat structure lead to information-geometric extensions of the Helmholtz-Hodge decomposition and the Otto structure in $L^{2}$ Wasserstein geometry. The structure can be extended to non-gradient nonequilibrium flows, from which we also obtain the induced dually flat structure on cycle spaces. This abstract but general framework can extend the applicability of information geometry to various problems of linear and nonlinear dynamics.

Information Theory,Differential Geometry,Statistics Theory,Chemical Physics,Data Analysis, Statistics and Probability
Stochastic Geometric Optimization with Joint Probabilistic Constraints

Jia Liu,Abdel Lisser,Zhiping Chen

DOI: https://doi.org/10.1016/j.orl.2016.08.002

IF: 1.151

2016-01-01

Operations Research Letters

Abstract:This paper discusses geometric programs with joint probabilistic constraints. When the stochastic parameters are normally distributed and independent of each other, we approximate the problem by using piecewise linear functions, and transform the approximation problem into a convex geometric program. We prove that this approximation method provides a lower bound. Then, we design a sequential convex optimization algorithm to find an upper bound. Finally, numerical tests are carried out on a stochastic shape optimization problem.
Geometry of Matrix Decompositions Seen Through Optimal Transport and Information Geometry

Klas Modin

DOI: https://doi.org/10.3934/jgm.2017014

2017-11-18

Abstract:The space of probability densities is an infinite-dimensional Riemannian manifold, with Riemannian metrics in two flavors: Wasserstein and Fisher--Rao. The former is pivotal in optimal mass transport (OMT), whereas the latter occurs in information geometry---the differential geometric approach to statistics. The Riemannian structures restrict to the submanifold of multivariate Gaussian distributions, where they induce Riemannian metrics on the space of covariance matrices. Here we give a systematic description of classical matrix decompositions (or factorizations) in terms of Riemannian geometry and compatible principal bundle structures. Both Wasserstein and Fisher--Rao geometries are discussed. The link to matrices is obtained by considering OMT and information geometry in the category of linear transformations and multivariate Gaussian distributions. This way, OMT is directly related to the polar decomposition of matrices, whereas information geometry is directly related to the $QR$, Cholesky, spectral, and singular value decompositions. We also give a coherent description of gradient flow equations for the various decompositions; most flows are illustrated in numerical examples. The paper is a combination of previously known and original results. As a survey it covers the Riemannian geometry of OMT and polar decompositions (smooth and linear category), entropy gradient flows, and the Fisher--Rao metric and its geodesics on the statistical manifold of multivariate Gaussian distributions. The original contributions include new gradient flows associated with various matrix decompositions, new geometric interpretations of previously studied isospectral flows, and a new proof of the polar decomposition of matrices based an entropy gradient flow.

Differential Geometry,Numerical Analysis
Information geometry of Markov Kernels: a survey

Geoffrey Wolfer,Shun Watanabe

DOI: https://doi.org/10.3389/fphy.2023.1195562

IF: 3.718

2023-07-28

Frontiers in Physics

Abstract:Information geometry and Markov chains are two powerful tools used in modern fields such as finance, physics, computer science, and epidemiology. In this survey, we explore their intersection, focusing on the theoretical framework. We attempt to provide a self-contained treatment of the foundations without requiring a solid background in differential geometry. We present the core concepts of information geometry of Markov chains, including information projections and the pivotal information geometric construction of Nagaoka. We then delve into recent advances in the field, such as geometric structures arising from time reversibility, lumpability of Markov chains, or tree models. Finally, we highlight practical applications of this framework, such as parameter estimation, hypothesis testing, large deviation theory, and the maximum entropy principle.

physics, multidisciplinary
Examples of Application of Nonparametric Information Geometry to Statistical Physics

Giovanni Pistone

DOI: https://doi.org/10.48550/arXiv.1308.5312

2013-08-24

Statistics Theory

Abstract:We review a nonparametric version of Amari's Information Geometry in which the set of positive probability densities on a given sample space is endowed with an atlas of charts to form a differentiable manifold modeled on Orlicz Banach spaces. This nonparametric setting is used to discuss the setting of typical problems in Machine Learning and Statistical Physics, such as relaxed optimization, Kullback-Leibler divergence, Boltzmann entropy, Boltzmann equation
Geometric Modeling in Probability and Statistics

Ovidiu Calin,Constantin Udrişte

DOI: https://doi.org/10.1007/978-3-319-07779-6

2014-01-01

Abstract:This book covers topics of Informational Geometry, a field which deals with the differential geometric study of the manifold probability density functions. This is a field that is increasingly attracting the interest of researchers from many different areas of science, including mathematics, statistics, geometry, computer science, signal processing, physics and neuroscience. It is the authors’ hope that the present book will be a valuable reference for researchers and graduate students in one of the aforementioned fields. This textbook is a unified presentation of differential geometry and probability theory, and constitutes a text for a course directed at graduate or advanced undergraduate students interested in applications of differential geometry in probability and statistics. The book contains over 100 proposed exercises meant to help students deepen their understanding, and it is accompanied by software that is able to provide numerical computations of several information geometric objects. The reader will understand a flourishing field of mathematics in which very few books have been written so far.
Information geometric methods for complexity

D. Felice,C. Cafaro,S. Mancini

DOI: https://doi.org/10.1063/1.5018926

2018-03-20

Abstract:Research on the use of information geometry (IG) in modern physics has witnessed significant advances recently. In this review article, we report on the utilization of IG methods to define measures of complexity in both classical and, whenever available, quantum physical settings. A paradigmatic example of a dramatic change in complexity is given by phase transitions (PTs). Hence we review both global and local aspects of PTs described in terms of the scalar curvature of the parameter manifold and the components of the metric tensor, respectively. We also report on the behavior of geodesic paths on the parameter manifold used to gain insight into the dynamics of PTs. Going further, we survey measures of complexity arising in the geometric framework. In particular, we quantify complexity of networks in terms of the Riemannian volume of the parameter space of a statistical manifold associated with a given network. We are also concerned with complexity measures that account for the interactions of a given number of parts of a system that cannot be described in terms of a smaller number of parts of the system. Finally, we investigate complexity measures of entropic motion on curved statistical manifolds that arise from a probabilistic description of physical systems in the presence of limited information. The Kullback-Leibler divergence, the distance to an exponential family and volumes of curved parameter manifolds, are examples of essential IG notions exploited in our discussion of complexity. We conclude by discussing strengths, limits, and possible future applications of IG methods to the physics of complexity.

Statistical Mechanics,Mathematical Physics
GAIT: A Geometric Approach to Information Theory

Jose Gallego-Posada,Ankit Vani,Max Schwarzer,Simon Lacoste-Julien

DOI: https://doi.org/10.48550/arXiv.1906.08325

2020-10-14

Abstract:We advocate the use of a notion of entropy that reflects the relative abundances of the symbols in an alphabet, as well as the similarities between them. This concept was originally introduced in theoretical ecology to study the diversity of ecosystems. Based on this notion of entropy, we introduce geometry-aware counterparts for several concepts and theorems in information theory. Notably, our proposed divergence exhibits performance on par with state-of-the-art methods based on the Wasserstein distance, but enjoys a closed-form expression that can be computed efficiently. We demonstrate the versatility of our method via experiments on a broad range of domains: training generative models, computing image barycenters, approximating empirical measures and counting modes.

Machine Learning
An Information-Theoretic Framework for Out-of-Distribution Generalization with Applications to Stochastic Gradient Langevin Dynamics

Wenliang Liu,Guanding Yu,Lele Wang,Renjie Liao

2024-12-13

Abstract:We study the Out-of-Distribution (OOD) generalization in machine learning and propose a general framework that establishes information-theoretic generalization bounds. Our framework interpolates freely between Integral Probability Metric (IPM) and $f$-divergence, which naturally recovers some known results (including Wasserstein- and KL-bounds), as well as yields new generalization bounds. Additionally, we show that our framework admits an optimal transport interpretation. When evaluated in two concrete examples, the proposed bounds either strictly improve upon existing bounds in some cases or match the best existing OOD generalization bounds. Moreover, by focusing on $f$-divergence and combining it with the Conditional Mutual Information (CMI) methods, we derive a family of CMI-based generalization bounds, which include the state-of-the-art ICIMI bound as a special instance. Finally, leveraging these findings, we analyze the generalization of the Stochastic Gradient Langevin Dynamics (SGLD) algorithm, showing that our derived generalization bounds outperform existing information-theoretic generalization bounds in certain scenarios.

Information Theory,Machine Learning
Optimization Algorithms for Faster Computational Geometry.

Zeyuan Allen-Zhu,Zhenyu Liao,Yang Yuan

DOI: https://doi.org/10.4230/lipics.icalp.2016.53

2016-01-01

Abstract:We study two fundamental problems in computational geometry: finding the maximum inscribed ball (MaxIB) inside a bounded polyhedron defined by $m$ hyperplanes, and the minimum enclosing ball (MinEB) of a set of $n$ points, both in $d$-dimensional space. We improve the running time of iterative algorithms on MaxIB from $\tilde{O}(m d \alpha^3 / \varepsilon^3)$ to $\tilde{O}(md + m \sqrt{d} \alpha / \varepsilon)$, a speed-up up to $\tilde{O}(\sqrt{d} \alpha^2 / \varepsilon^2)$, and MinEB from $\tilde{O}(n d / \sqrt{\varepsilon})$ to $\tilde{O}(nd + n \sqrt{d} / \sqrt{\varepsilon})$, a speed-up up to $\tilde{O}(\sqrt{d})$. Our improvements are based on a novel saddle-point optimization framework. We propose a new algorithm $\mathtt{L1L2SPSolver}$ for solving a class of regularized saddle-point problems, and apply a randomized Hadamard space rotation which is a technique borrowed from compressive sensing. Interestingly, the motivation of using Hadamard rotation solely comes from our optimization view but not the original geometry problem: indeed, it is not immediately clear why MaxIB or MinEB, as a geometric problem, should be easier to solve if we rotate the space by a unitary matrix. We hope that our optimization perspective sheds lights on solving other geometric problems as well.
Methods for Optimization Problems with Markovian Stochasticity and Non-Euclidean Geometry

Vladimir Solodkin,Andrew Veprikov,Aleksandr Beznosikov

2024-08-04

Abstract:This paper examines a variety of classical optimization problems, including well-known minimization tasks and more general variational inequalities. We consider a stochastic formulation of these problems, and unlike most previous work, we take into account the complex Markov nature of the noise. We also consider the geometry of the problem in an arbitrary non-Euclidean setting, and propose four methods based on the Mirror Descent iteration technique. Theoretical analysis is provided for smooth and convex minimization problems and variational inequalities with Lipschitz and monotone operators. The convergence guarantees obtained are optimal for first-order stochastic methods, as evidenced by the lower bound estimates provided in this paper.

Optimization and Control
Geometric thermodynamics for the Fokker-Planck equation: Stochastic thermodynamic links between information geometry and optimal transport

Sosuke Ito

DOI: https://doi.org/10.1007/s41884-023-00102-3

2023-12-13

Abstract:We propose a geometric theory of non-equilibrium thermodynamics, namely geometric thermodynamics, using our recent developments of differential-geometric aspects of entropy production rate in non-equilibrium thermodynamics. By revisiting our recent results on geometrical aspects of entropy production rate in stochastic thermodynamics for the Fokker-Planck equation, we introduce a geometric framework of non-equilibrium thermodynamics in terms of information geometry and optimal transport theory. We show that the proposed geometric framework is useful for obtaining several non-equilibrium thermodynamic relations, such as thermodynamic trade-off relations between the thermodynamic cost and the fluctuation of the observable, optimal protocols for the minimum thermodynamic cost and the decomposition of the entropy production rate for the non-equilibrium system. We clarify several stochastic-thermodynamic links between information geometry and optimal transport theory via the excess entropy production rate based on a relation between the gradient flow expression and information geometry in the space of probability densities and a relation between the velocity field in optimal transport and information geometry in the space of path probability densities.

Statistical Mechanics,Mathematical Physics
Algorithm-independent bounds on complex optimization through the statistics of marginal optima

Jaron Kent-Dobias

2024-07-24

Abstract:Optimization seeks extremal points in a function. When there are superextensively many optima, optimization algorithms are liable to get stuck. Under these conditions, generic algorithms tend to find marginal optima, which have many nearly flat directions. In a companion paper, we introduce a technique to count marginal optima in random landscapes. Here, we use the statistics of marginal optima calculated using this technique to produce generic bounds on optimization, based on the simple principle that algorithms will overwhelmingly tend to get stuck only where marginal optima are found. We demonstrate the idea on a simple non-Gaussian problem of maximizing the sum of squared random functions on a compact space. Numeric experiments using both gradient descent and generalized approximate message passing algorithms fall inside the expected bounds.

Disordered Systems and Neural Networks,Statistical Mechanics
Information Geometry of the Gaussian Space

Giovanni Pistone

DOI: https://doi.org/10.48550/arXiv.1803.08135

2018-03-20

Probability

Abstract:We discuss the Pistone-Sempi exponential manifold on the finite-dimensional Gaussian space. We consider the role of the entropy, the continuity of translations, Poincar\'e-type inequalities, the generalized differentiability of probability densities of the Gaussian space.

Optimization via Information Geometry

Information Geometry for the Working Information Theorist

Progress in Theory and Applications of Information Geometry

Information Geometry of the Probability Simplex: A Short Course

Geometry, Computation, and Optimality in Stochastic Optimization

An Invariant Information Geometric Method for High-Dimensional Online Optimization

Dually affine Information Geometry modeled on a Banach space

Information Geometry of Dynamics on Graphs and Hypergraphs

Stochastic Geometric Optimization with Joint Probabilistic Constraints

Geometry of Matrix Decompositions Seen Through Optimal Transport and Information Geometry

Information geometry of Markov Kernels: a survey

Examples of Application of Nonparametric Information Geometry to Statistical Physics

Geometric Modeling in Probability and Statistics

Information geometric methods for complexity

GAIT: A Geometric Approach to Information Theory

An Information-Theoretic Framework for Out-of-Distribution Generalization with Applications to Stochastic Gradient Langevin Dynamics

Optimization Algorithms for Faster Computational Geometry.

Methods for Optimization Problems with Markovian Stochasticity and Non-Euclidean Geometry

Geometric thermodynamics for the Fokker-Planck equation: Stochastic thermodynamic links between information geometry and optimal transport

Algorithm-independent bounds on complex optimization through the statistics of marginal optima

Information Geometry of the Gaussian Space