Abstract:The ability to compute the exact divergence between two high-dimensional distributions is useful in many applications but doing so naively is intractable. Computing the alpha-beta divergence -- a family of divergences that includes the Kullback-Leibler divergence and Hellinger distance -- between the joint distribution of two decomposable models, i.e chordal Markov networks, can be done in time exponential in the treewidth of these models. However, reducing the dissimilarity between two high-dimensional objects to a single scalar value can be uninformative. Furthermore, in applications such as supervised learning, the divergence over a conditional distribution might be of more interest. Therefore, we propose an approach to compute the exact alpha-beta divergence between any marginal or conditional distribution of two decomposable models. Doing so tractably is non-trivial as we need to decompose the divergence between these distributions and therefore, require a decomposition over the marginal and conditional distributions of these models. Consequently, we provide such a decomposition and also extend existing work to compute the marginal and conditional alpha-beta divergence between these decompositions. We then show how our method can be used to analyze distributional changes by first applying it to a benchmark image dataset. Finally, based on our framework, we propose a novel way to quantify the error in contemporary superconducting quantum computers. Code for all experiments is available at: https://lklee.dev/pub/2023-icdm/code

Upper and lower bounds for approximation of the Kullback-Leibler divergence between Hidden Markov models

Variational Approach for Efficient KL Divergence Estimation in Dirichlet Mixture Models

Bayesian estimation of the Kullback-Leibler divergence for categorical sytems using mixtures of Dirichlet priors

A New Distance Measure for Hidden Markov Models

Exact Computation of Kullback-Leibler Distance for Hidden Markov Trees and Models

Bayesian estimation of the Kullback-Leibler divergence for categorical systems using mixtures of Dirichlet priors

Process Monitoring Based on Kullback Leibler Divergence

Approximate Optimal Estimation Based on Kullback–Leibler Divergence for Lossy Networks Without Acknowledgement

Minimum Kullback–Leibler Divergence Parameter Generation for HMM-Based Speech Synthesis

Kullback-Leibler Divergence and Akaike Information Criterion in General Hidden Markov Models

Exact Expressions for Kullback–Leibler Divergence for Univariate Distributions

Computing Marginal and Conditional Divergences between Decomposable Models with Applications

Risk Bounds for Mixture Density Estimation on Compact Domains via the $h$-Lifted Kullback--Leibler Divergence

Taming numerical imprecision by adapting the KL divergence to negative probabilities

co-BPM: a Bayesian Model for Divergence Estimation

Decoupled Kullback-Leibler Divergence Loss

HMM-Based Hierarchical Unit Selection Combining Kullback-Leibler Divergence with Likelihood Criterion

A data driven method for target and concatenation cost calculation with KL-Divergence in Mandarin hybrid speech synthesis

Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space

A New Minimum Divergence Approach To Discriminative Training

Amplifying Inter-Message Distance: On Information Divergence Measures in Big Data.