Abstract:Statistical divergences quantify the difference between probability distributions, thereby allowing for multiple uses in machine-learning. However, a fundamental challenge of these quantities is their estimation from empirical samples since the underlying distributions of the data are usually unknown. In this work, we propose a divergence inspired by the Jensen-Shannon divergence which avoids the estimation of the probability density functions. Our approach embeds the data in an reproducing kernel Hilbert space (RKHS) where we associate data distributions with uncentered covariance operators in this representation space. Therefore, we name this measure the representation Jensen-Shannon divergence (RJSD). We provide an estimator from empirical covariance matrices by explicitly mapping the data to an RKHS using Fourier features. This estimator is flexible, scalable, differentiable, and suitable for minibatch-based optimization problems. Additionally, we provide an estimator based on kernel matrices without an explicit mapping to the RKHS. We provide consistency convergence results for the proposed estimator. Moreover, we demonstrate that this quantity is a lower bound on the Jensen-Shannon divergence, leading to a variational approach to estimate it with theoretical guarantees. We leverage the proposed divergence to train generative networks, where our method mitigates mode collapse and encourages samples diversity. Additionally, RJSD surpasses other state-of-the-art techniques in multiple two-sample testing problems, demonstrating superior performance and reliability in discriminating between distributions.

Exact Expressions for Kullback-Leibler Divergence for Multivariate and Matrix-Variate Distributions

Exact Expressions for Kullback–Leibler Divergence for Univariate Distributions

Detecting abnormal situations using the Kullback-Leibler divergence.

Kullback-Leibler divergence for the Fréchet extreme-value distribution

On the Kullback-Leibler divergence between discrete normal distributions

Central Limit Theorem on Symmetric Kullback-Leibler (KL) Divergence

Codivergences and information matrices

Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence

The Logarithmic Super Divergence and its use in Statistical Inference

Log-Determinant Divergences Revisited: Alpha-Beta and Gamma Log-Det Divergences

Divergence measures based on the Shannon entropy

Limit Distribution for Smooth Total Variation and $χ^2$-Divergence in High Dimensions

Bayesian estimation of the Kullback-Leibler divergence for categorical systems using mixtures of Dirichlet priors

Bayesian estimation of the Kullback-Leibler divergence for categorical sytems using mixtures of Dirichlet priors

The $q$-generalised multinomial/KL-divergence correspondence

The Representation Jensen-Shannon Divergence

Minimal Kullback–Leibler Divergence for Constrained Lévy–Itô Processes

Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space

Divergence Measure of Belief Function and Its Application in Data Fusion

Transport information Bregman divergences

Sum-of-Squares Relaxations for Information Theory and Variational Inference