Abstract:The ability to compute the exact divergence between two high-dimensional distributions is useful in many applications but doing so naively is intractable. Computing the alpha-beta divergence -- a family of divergences that includes the Kullback-Leibler divergence and Hellinger distance -- between the joint distribution of two decomposable models, i.e chordal Markov networks, can be done in time exponential in the treewidth of these models. However, reducing the dissimilarity between two high-dimensional objects to a single scalar value can be uninformative. Furthermore, in applications such as supervised learning, the divergence over a conditional distribution might be of more interest. Therefore, we propose an approach to compute the exact alpha-beta divergence between any marginal or conditional distribution of two decomposable models. Doing so tractably is non-trivial as we need to decompose the divergence between these distributions and therefore, require a decomposition over the marginal and conditional distributions of these models. Consequently, we provide such a decomposition and also extend existing work to compute the marginal and conditional alpha-beta divergence between these decompositions. We then show how our method can be used to analyze distributional changes by first applying it to a benchmark image dataset. Finally, based on our framework, we propose a novel way to quantify the error in contemporary superconducting quantum computers. Code for all experiments is available at: https://lklee.dev/pub/2023-icdm/code

Generalizing Jensen and Bregman divergences with comparative convexity and the statistical Bhattacharyya distances with comparable means

Divergences Induced by the Cumulant and Partition Functions of Exponential Families and Their Deformations Induced by Comparative Convexity

Divergences induced by dual subtractive and divisive normalizations of exponential families and their convex deformations

Generalized Arithmetic and Geometric Mean Divergence Measure and their Statistical Aspects

The Logarithmic Super Divergence and its use in Statistical Inference

On the Equivalence of Statistical Distances for Isotropic Convex Measures

The duo Fenchel-Young divergence

Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means

On the Kullback-Leibler divergence between discrete normal distributions

New Brunn--Minkowski and functional inequalities via convexity of entropy

The Bregman Distance, Approximate Compactness and Convexity of Chebyshev Sets in Banach Spaces

A Note on Statistical Distances for Discrete Log-Concave Measures

Symplectic Bregman divergences

Relative Density-Ratio Estimation for Robust Distribution Comparison

Divergence for s-concave and log concave functions

A Unifying Framework for Some Directed Distances in Statistics

Computing Marginal and Conditional Divergences between Decomposable Models with Applications

Further improvements of the Jensen inequality in the integral sense by virtue of 6-convexity along with applications

Codivergences and information matrices

On the f-divergences between densities of a multivariate location or scale family

Comparing Comparators in Generalization Bounds