The Information Geometry of UMAP

Alexander Kolpakov,Aidan Rocke

2024-06-03

Abstract:Although UMAP was derived from Category Theory observations, its underlying mechanisms may be clarified using Information Geometry.

Computational Geometry,Discrete Mathematics,Information Theory,Geometric Topology

What problem does this paper attempt to address?

Information Geometry for the Working Information Theorist

Kumar Vijay Mishra,M. Ashok Kumar,Ting-Kam Leonard Wong

2023-10-06

Abstract:Information geometry is a study of statistical manifolds, that is, spaces of probability distributions from a geometric perspective. Its classical information-theoretic applications relate to statistical concepts such as Fisher information, sufficient statistics, and efficient estimators. Today, information geometry has emerged as an interdisciplinary field that finds applications in diverse areas such as radar sensing, array signal processing, quantum physics, deep learning, and optimal transport. This article presents an overview of essential information geometry to initiate an information theorist, who may be unfamiliar with this exciting area of research. We explain the concepts of divergences on statistical manifolds, generalized notions of distances, orthogonality, and geodesics, thereby paving the way for concrete applications and novel theoretical investigations. We also highlight some recent information-geometric developments, which are of interest to the broader information theory community.

Information Theory,Machine Learning,Signal Processing,Differential Geometry
Uniform Manifold Approximation and Projection (UMAP) and its Variants: Tutorial and Survey

Benyamin Ghojogh,Ali Ghodsi,Fakhri Karray,Mark Crowley

DOI: https://doi.org/10.48550/arXiv.2109.02508

2021-08-25

Abstract:Uniform Manifold Approximation and Projection (UMAP) is one of the state-of-the-art methods for dimensionality reduction and data visualization. This is a tutorial and survey paper on UMAP and its variants. We start with UMAP algorithm where we explain probabilities of neighborhood in the input and embedding spaces, optimization of cost function, training algorithm, derivation of gradients, and supervised and semi-supervised embedding by UMAP. Then, we introduce the theory behind UMAP by algebraic topology and category theory. Then, we introduce UMAP as a neighbor embedding method and compare it with t-SNE and LargeVis algorithms. We discuss negative sampling and repulsive forces in UMAP's cost function. DensMAP is then explained for density-preserving embedding. We then introduce parametric UMAP for embedding by deep learning and progressive UMAP for streaming and out-of-sample data embedding.

Human-Computer Interaction,Machine Learning
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

Leland McInnes,John Healy,James Melville

DOI: https://doi.org/10.48550/arXiv.1802.03426

2020-09-18

Abstract:UMAP (Uniform Manifold Approximation and Projection) is a novel manifold learning technique for dimension reduction. UMAP is constructed from a theoretical framework based in Riemannian geometry and algebraic topology. The result is a practical scalable algorithm that applies to real world data. The UMAP algorithm is competitive with t-SNE for visualization quality, and arguably preserves more of the global structure with superior run time performance. Furthermore, UMAP has no computational restrictions on embedding dimension, making it viable as a general purpose dimension reduction technique for machine learning.

Machine Learning,Computational Geometry
Fuzzy simplicial sets and their application to geometric data analysis

Lukas Silvester Barth,Fatemeh,Fahimi,Parvaneh Joharinad,Jürgen Jost,Janis Keck,Thomas Jan Mikhail

2024-06-17

Abstract:In this article, we expand upon the concepts introduced in \cite{Spivak09} about the relationship between the category $\mathbf{UM}$ of uber metric spaces and the category $\mathbf{sFuz}$ of fuzzy simplicial sets. We show that fuzzy simplicial sets can be regarded as natural combinatorial generalizations of metric relations. Furthermore, we take inspiration from UMAP to apply the theory to dimension reduction (manifold learning) and data visualization, while refining some of their constructions to put the corresponding theory on a more solid footing. A generalization of the adjunction between $\mathbf{UM}$ and $\mathbf{sFuz}$ will allow us to view the adjunctions used in both publications as special cases. Moreover, we derive an explicit description of colimits in $\mathbf{UM}$ and the realization functor $\text{Re}:\mathbf{sFuz}\to\mathbf{UM}$, as well as rigorous definitions of functors that make it possible to recursively merge sets of fuzzy simplicial sets. We show that $\mathbf{UM}$ and the category of extended-pseudo metric spaces $\mathbf{EPMet}$ can be embedded into $\mathbf{sFuz}$ and provide a description of the adjunctions between the category of truncated fuzzy simplicial sets and $\mathbf{sFuz}$, which we relate to persistent homology. Combining those constructions, we can show a surprising connection between the well-known dimension reduction methods UMAP and Isomap and derive an alternative algorithm, which we call IsUMap, that combines some of the strengths of both methods. We compare it with UMAP and Isomap and provide explanations for observed differences.

Algebraic Topology,Category Theory,Metric Geometry
Geometry of Information Integration

Shun-ichi Amari,Naotsugu Tsuchiya,Masafumi Oizumi

DOI: https://doi.org/10.48550/arXiv.1709.02050

2017-09-07

Abstract:Information geometry is used to quantify the amount of information integration within multiple terminals of a causal dynamical system. Integrated information quantifies how much information is lost when a system is split into parts and information transmission between the parts is removed. Multiple measures have been proposed as a measure of integrated information. Here, we analyze four of the previously proposed measures and elucidate their relations from a viewpoint of information geometry. Two of them use dually flat manifolds and the other two use curved manifolds to define a split model. We show that there are hierarchical structures among the measures. We provide explicit expressions of these measures.

Information Theory
The basics of information geometry

Ariel Caticha

DOI: https://doi.org/10.1063/1.4905960

2015-01-01

AIP Conference Proceedings

Abstract:To what extent can we distinguish one probability distribution from another? Are there quantitative measures of distinguishability? The goal of this tutorial is to approach such questions by introducing the notion of the “distance” between two probability distributions and exploring some basic ideas of such an “information geometry”.
Fuzzy Information Discrimination Measures and Their Application to Low Dimensional Embedding Construction in the UMAP Algorithm

Liliya A Demidova,Artyom V Gorchakov

DOI: https://doi.org/10.3390/jimaging8040113

2022-04-15

Abstract:Dimensionality reduction techniques are often used by researchers in order to make high dimensional data easier to interpret visually, as data visualization is only possible in low dimensional spaces. Recent research in nonlinear dimensionality reduction introduced many effective algorithms, including t-distributed stochastic neighbor embedding (t-SNE), uniform manifold approximation and projection (UMAP), dimensionality reduction technique based on triplet constraints (TriMAP), and pairwise controlled manifold approximation (PaCMAP), aimed to preserve both the local and global structure of high dimensional data while reducing the dimensionality. The UMAP algorithm has found its application in bioinformatics, genetics, genomics, and has been widely used to improve the accuracy of other machine learning algorithms. In this research, we compare the performance of different fuzzy information discrimination measures used as loss functions in the UMAP algorithm while constructing low dimensional embeddings. In order to achieve this, we derive the gradients of the considered losses analytically and employ the Adam algorithm during the loss function optimization process. From the conducted experimental studies we conclude that the use of either the logarithmic fuzzy cross entropy loss without reduced repulsion or the symmetric logarithmic fuzzy cross entropy loss with sufficiently large neighbor count leads to better global structure preservation of the original multidimensional data when compared to the loss function used in the original UMAP algorithm implementation.
Information geometry of Markov Kernels: a survey

Geoffrey Wolfer,Shun Watanabe

DOI: https://doi.org/10.3389/fphy.2023.1195562

IF: 3.718

2023-07-28

Frontiers in Physics

Abstract:Information geometry and Markov chains are two powerful tools used in modern fields such as finance, physics, computer science, and epidemiology. In this survey, we explore their intersection, focusing on the theoretical framework. We attempt to provide a self-contained treatment of the foundations without requiring a solid background in differential geometry. We present the core concepts of information geometry of Markov chains, including information projections and the pivotal information geometric construction of Nagaoka. We then delve into recent advances in the field, such as geometric structures arising from time reversibility, lumpability of Markov chains, or tree models. Finally, we highlight practical applications of this framework, such as parameter estimation, hypothesis testing, large deviation theory, and the maximum entropy principle.

physics, multidisciplinary
On UMAP's true loss function

Sebastian Damrich,Fred A. Hamprecht

DOI: https://doi.org/10.48550/arXiv.2103.14608

2021-04-22

Abstract:UMAP has supplanted t-SNE as state-of-the-art for visualizing high-dimensional datasets in many disciplines, but the reason for its success is not well understood. In this work, we investigate UMAP's sampling based optimization scheme in detail. We derive UMAP's effective loss function in closed form and find that it differs from the published one. As a consequence, we show that UMAP does not aim to reproduce its theoretically motivated high-dimensional UMAP similarities. Instead, it tries to reproduce similarities that only encode the shared $k$ nearest neighbor graph, thereby challenging the previous understanding of UMAP's effectiveness. Instead, we claim that the key to UMAP's success is its implicit balancing of attraction and repulsion resulting from negative sampling. This balancing in turn facilitates optimization via gradient descent. We corroborate our theoretical findings on toy and single cell RNA sequencing data.

Machine Learning
Optimization via Information Geometry

Luigi Malagò,Giovanni Pistone

DOI: https://doi.org/10.48550/arXiv.1403.3762

2014-03-15

Statistics Theory

Abstract:Information Geometry has been used to inspire efficient algorithms for stochastic optimization, both in the combinatorial and the continuous case. We give an overview of the authors' research program and some specific contributions to the underlying theory.
The EM Algorithm in Information Geometry

Sammy Suliman

2024-05-01

Abstract:The purpose of this thesis is to convey the basic concepts of information geometry and its applications to non-specialists and those in applied fields, assuming only a first-year undergraduate background in calculus, linear algebra, and probability theory / statistics. We first begin with an introduction to the EM algorithm, providing a typical use case in Python, before moving to an overview of basic Riemannian geometry. We then introduce the core concepts of information geometry and the $em$ algorithm, with an explicit calculation of both the $e$ and $m$ projection, before closing with a discussion of an important application of this research to the field of deep learning, providing a novel implementation in Python.

History and Overview,Information Theory
Information Geometry and Statistical Manifold

Mashbat Suzuki

DOI: https://doi.org/10.48550/arXiv.1410.3369

2014-10-09

Abstract:We review basic notions in the field of information geometry such as Fisher metric on statistical manifold, $\alpha$-connection and corresponding curvature following Amari's work . We show application of information geometry to asymptotic statistical inference.

Statistics Theory,Differential Geometry,Probability
Parametric UMAP embeddings for representation and semi-supervised learning

Tim Sainburg,Leland McInnes,Timothy Q Gentner

DOI: https://doi.org/10.48550/arXiv.2009.12981

2021-08-30

Abstract:UMAP is a non-parametric graph-based dimensionality reduction algorithm using applied Riemannian geometry and algebraic topology to find low-dimensional embeddings of structured data. The UMAP algorithm consists of two steps: (1) Compute a graphical representation of a dataset (fuzzy simplicial complex), and (2) Through stochastic gradient descent, optimize a low-dimensional embedding of the graph. Here, we extend the second step of UMAP to a parametric optimization over neural network weights, learning a parametric relationship between data and embedding. We first demonstrate that Parametric UMAP performs comparably to its non-parametric counterpart while conferring the benefit of a learned parametric mapping (e.g. fast online embeddings for new data). We then explore UMAP as a regularization, constraining the latent distribution of autoencoders, parametrically varying global structure preservation, and improving classifier accuracy for semi-supervised learning by capturing structure in unlabeled data. Google Colab walkthrough: <a class="link-external link-https" href="https://colab.research.google.com/drive/1WkXVZ5pnMrm17m0YgmtoNjM_XHdnE5Vp?usp=sharing" rel="external noopener nofollow">this https URL</a>

Machine Learning,Computational Geometry,Quantitative Methods
Understanding Higher-Order Interactions in Information Space

Herbert Edelsbrunner,Katharina Ölsböck,Hubert Wagner

DOI: https://doi.org/10.3390/e26080637

2024-07-27

Abstract:Methods used in topological data analysis naturally capture higher-order interactions in point cloud data embedded in a metric space. This methodology was recently extended to data living in an information space, by which we mean a space measured with an information theoretical distance. One such setting is a finite collection of discrete probability distributions embedded in the probability simplex measured with the relative entropy (Kullback-Leibler divergence). More generally, one can work with a Bregman divergence parameterized by a different notion of entropy. While theoretical algorithms exist for this setup, there is a paucity of implementations for exploring and comparing geometric-topological properties of various information spaces. The interest of this work is therefore twofold. First, we propose the first robust algorithms and software for geometric and topological data analysis in information space. Perhaps surprisingly, despite working with Bregman divergences, our design reuses robust libraries for the Euclidean case. Second, using the new software, we take the first steps towards understanding the geometric-topological structure of these spaces. In particular, we compare them with the more familiar spaces equipped with the Euclidean and Fisher metrics.
Geometry of Matrix Decompositions Seen Through Optimal Transport and Information Geometry

Klas Modin

DOI: https://doi.org/10.3934/jgm.2017014

2017-11-18

Abstract:The space of probability densities is an infinite-dimensional Riemannian manifold, with Riemannian metrics in two flavors: Wasserstein and Fisher--Rao. The former is pivotal in optimal mass transport (OMT), whereas the latter occurs in information geometry---the differential geometric approach to statistics. The Riemannian structures restrict to the submanifold of multivariate Gaussian distributions, where they induce Riemannian metrics on the space of covariance matrices. Here we give a systematic description of classical matrix decompositions (or factorizations) in terms of Riemannian geometry and compatible principal bundle structures. Both Wasserstein and Fisher--Rao geometries are discussed. The link to matrices is obtained by considering OMT and information geometry in the category of linear transformations and multivariate Gaussian distributions. This way, OMT is directly related to the polar decomposition of matrices, whereas information geometry is directly related to the $QR$, Cholesky, spectral, and singular value decompositions. We also give a coherent description of gradient flow equations for the various decompositions; most flows are illustrated in numerical examples. The paper is a combination of previously known and original results. As a survey it covers the Riemannian geometry of OMT and polar decompositions (smooth and linear category), entropy gradient flows, and the Fisher--Rao metric and its geodesics on the statistical manifold of multivariate Gaussian distributions. The original contributions include new gradient flows associated with various matrix decompositions, new geometric interpretations of previously studied isospectral flows, and a new proof of the polar decomposition of matrices based an entropy gradient flow.

Differential Geometry,Numerical Analysis
Dually affine Information Geometry modeled on a Banach space

Goffredo Chirco,Giovanni Pistone

2024-05-13

Abstract:In this chapter, we study Information Geometry from a particular non-parametric or functional point of view. The basic model is a probabilities subset usually specified by regularity conditions. For example, probability measures mutually absolutely continuous or probability densities with a given degree of smoothness. We construct a manifold structure by giving an atlas of charts as mappings from probabilities to a Banach space. The charts we use are quite peculiar in that we consider only instances where the transition mappings are affine. We chose a particular expression of the tangent and cotangent bundles in this affine setting.

Statistics Theory
Information geometry of diffeomorphism groups

Boris Khesin,Gerard Misiołek,Klas Modin

2024-11-06

Abstract:The study of diffeomorphism groups and their applications to problems in analysis and geometry has a long history. In geometric hydrodynamics, pioneered by V.~Arnold in the 1960s, one considers an ideal fluid flow as the geodesic motion on the infinite-dimensional group of volume-preserving diffeomorphisms of the fluid domain with respect to the metric defined by the kinetic energy. Similar considerations on the space of densities lead to a geometric description of optimal mass transport and the Kantorovich-Wasserstein metric. Likewise, information geometry associated with the Fisher-Rao metric and the Hellinger distance has an equally beautiful infinite-dimensional geometric description and can be regarded as a higher-order Sobolev analogue of optimal transportation. In this work we review various metrics on diffeomorphism groups relevant to this approach and introduce appropriate topology, smooth structures and dynamics on the corresponding infinite-dimensional manifolds. Our main goal is to demonstrate how, alongside topological hydrodynamics, Hamiltonian dynamics and optimal mass transport, information geometry with its elaborate toolbox has become yet another exciting field for applications of geometric analysis on diffeomorphism groups.

Differential Geometry,Information Theory,Mathematical Physics
A unified framework for information integration based on information geometry

Masafumi Oizumi,Naotsugu Tsuchiya,Shun-ichi Amari

DOI: https://doi.org/10.1073/pnas.1603583113

2015-10-15

Abstract:We propose a unified theoretical framework for quantifying spatio-temporal interactions in a stochastic dynamical system based on information geometry. In the proposed framework, the degree of interactions is quantified by the divergence between the actual probability distribution of the system and a constrained probability distribution where the interactions of interest are disconnected. This framework provides novel geometric interpretations of various information theoretic measures of interactions, such as mutual information, transfer entropy, and stochastic interaction in terms of how interactions are disconnected. The framework therefore provides an intuitive understanding of the relationships between the various quantities. By extending the concept of transfer entropy, we propose a novel measure of integrated information which measures causal interactions between parts of a system. Integrated information quantifies the extent to which the whole is more than the sum of the parts and can be potentially used as a biological measure of the levels of consciousness.

Neurons and Cognition,Information Theory
Information Geometry of Wasserstein Statistics on Shapes and Affine Deformations

Shun-ichi Amari,Takeru Matsuda

2024-06-25

Abstract:Information geometry and Wasserstein geometry are two main structures introduced in a manifold of probability distributions, and they capture its different characteristics. We study characteristics of Wasserstein geometry in the framework of Li and Zhao (2023) for the affine deformation statistical model, which is a multi-dimensional generalization of the location-scale model. We compare merits and demerits of estimators based on information geometry and Wasserstein geometry. The shape of a probability distribution and its affine deformation are separated in the Wasserstein geometry, showing its robustness against the waveform perturbation in exchange for the loss in Fisher efficiency. We show that the Wasserstein estimator is the moment estimator in the case of the elliptically symmetric affine deformation model. It coincides with the information-geometrical estimator (maximum-likelihood estimator) when the waveform is Gaussian. The role of the Wasserstein efficiency is elucidated in terms of robustness against waveform change.

Statistics Theory,Machine Learning
Information Geometry of Dynamics on Graphs and Hypergraphs

Tetsuya J. Kobayashi,Dimitri Loutchko,Atsushi Kamimura,Shuhei A. Horiguchi,Yuki Sughiyama

2023-08-05

Abstract:We introduce a new information-geometric structure associated with the dynamics on discrete objects such as graphs and hypergraphs. The presented setup consists of two dually flat structures built on the vertex and edge spaces, respectively. The former is the conventional duality between density and potential, e.g., the probability density and its logarithmic form induced by a convex thermodynamic function. The latter is the duality between flux and force induced by a convex and symmetric dissipation function, which drives the dynamics of the density. These two are connected topologically by the homological algebraic relation induced by the underlying discrete objects. The generalized gradient flow in this doubly dual flat structure is an extension of the gradient flows on Riemannian manifolds, which include Markov jump processes and nonlinear chemical reaction dynamics as well as the natural gradient and mirror descent. The information-geometric projections on this doubly dual flat structure lead to information-geometric extensions of the Helmholtz-Hodge decomposition and the Otto structure in $L^{2}$ Wasserstein geometry. The structure can be extended to non-gradient nonequilibrium flows, from which we also obtain the induced dually flat structure on cycle spaces. This abstract but general framework can extend the applicability of information geometry to various problems of linear and nonlinear dynamics.

Information Theory,Differential Geometry,Statistics Theory,Chemical Physics,Data Analysis, Statistics and Probability

The Information Geometry of UMAP

Information Geometry for the Working Information Theorist

Uniform Manifold Approximation and Projection (UMAP) and its Variants: Tutorial and Survey

UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

Fuzzy simplicial sets and their application to geometric data analysis

Geometry of Information Integration

The basics of information geometry

Fuzzy Information Discrimination Measures and Their Application to Low Dimensional Embedding Construction in the UMAP Algorithm

Information geometry of Markov Kernels: a survey

On UMAP's true loss function

Optimization via Information Geometry

The EM Algorithm in Information Geometry

Information Geometry and Statistical Manifold

Parametric UMAP embeddings for representation and semi-supervised learning

Understanding Higher-Order Interactions in Information Space

Geometry of Matrix Decompositions Seen Through Optimal Transport and Information Geometry

Dually affine Information Geometry modeled on a Banach space

Information geometry of diffeomorphism groups

A unified framework for information integration based on information geometry

Information Geometry of Wasserstein Statistics on Shapes and Affine Deformations

Information Geometry of Dynamics on Graphs and Hypergraphs