Abstract:Living organisms rely on internal models of the world to act adaptively. These models cannot encode every detail and hence need to compress information. From a cognitive standpoint, information compression can manifest as a distortion of latent representations, resulting in the emergence of representations that may not accurately reflect the external world or its geometry. Rate-distortion theory formalizes the optimal way to compress information, by considering factors such as capacity limitations, the frequency and the utility of stimuli. However, while this theory explains why the above factors distort latent representations, it does not specify which specific distortions they produce. To address this question, here we systematically explore the geometry of the latent representations that emerge in generative models that operate under the principles of rate-distortion theory ($\beta$-VAEs). Our results highlight that three main classes of distortions of internal representations -- prototypization, specialization, orthogonalization -- emerge as signatures of information compression, under constraints on capacity, data distributions and tasks. These distortions can coexist, giving rise to a rich landscape of latent spaces, whose geometry could differ significantly across generative models subject to different constraints. Our findings contribute to explain how the normative constraints of rate-distortion theory distort the geometry of latent representations of generative models of artificial systems and living organisms.

Distortion in Correspondence Analysis and in Taxicab Correspondence Analysis: A Comparison

Distorted copulas

Distortion-Free Nonlinear Dimensionality Reduction

Visualization of Extremely Sparse Contingency Table by Taxicab Correspondence Analysis: A Case Study of Textual Data

Some notes on Goodman's marginal-free correspondence analysis

Quasiregular distortion of dimensions

The least Euclidean distortion constant of a distance-regular graph

Stochastic Distortion And Its Transformed Copula

Coherent Distorted Beliefs

On quasiconformal dimension distortion for subsets of the real line

A Geometric Unification of Distributionally Robust Covariance Estimators: Shrinking the Spectrum by Inflating the Ambiguity Set

The Perfect Marriage and Much More: Combining Dimension Reduction, Distance Measures and Covariance

Distorted optimal transport

High-Dimensional Canonical Correlation Analysis

Mappings of finite distortion on metric surfaces

Big Data Scaling through Metric Mapping: Exploiting the Remarkable Simplicity of Very High Dimensional Spaces using Correspondence Analysis

Dimensionality Reduction of Dynamics on Lie Manifolds via Structure-Aware Canonical Correlation Analysis

The geometry of efficient codes: how rate-distortion trade-offs distort the latent representations of generative models

The distortion principle for insurance pricing: properties, identification and robustness

Generalized Gini's mean difference through distortions and copulas, and related minimizing problems