On Bayesian estimation via divergences

Mohamed Cherfi

DOI: https://doi.org/10.1016/j.crma.2014.06.013

2014-09-01

Comptes Rendus Mathematique

Abstract:In this note, we introduce a new methodology for Bayesian inference through the use of ϕ-divergences and of the duality technique. The asymptotic laws of the estimates are established.

mathematics

What problem does this paper attempt to address?

Bayesian estimation of the Kullback-Leibler divergence for categorical systems using mixtures of Dirichlet priors

Francesco Camaglia,Ilya Nemenman,Thierry Mora,and Aleksandra M. Walczak

DOI: https://doi.org/10.1103/physreve.109.024305

IF: 2.707

2024-02-14

Physical Review E

Abstract:In many applications in biology, engineering, and economics, identifying similarities and differences between distributions of data from complex processes requires comparing finite categorical samples of discrete counts. Statistical divergences quantify the difference between two distributions. However, their estimation is very difficult and empirical methods often fail, especially when the samples are small. We develop a Bayesian estimator of the Kullback-Leibler divergence between two probability distributions that makes use of a mixture of Dirichlet priors on the distributions being compared. We study the properties of the estimator on two examples: probabilities drawn from Dirichlet distributions and random strings of letters drawn from Markov chains. We extend the approach to the squared Hellinger divergence. Both estimators outperform other estimation techniques, with better results for data with a large number of categories and for higher values of divergences. https://doi.org/10.1103/PhysRevE.109.024305 ©2024 American Physical Society

physics, fluids & plasmas, mathematical
Bayesian estimation of the Kullback-Leibler divergence for categorical sytems using mixtures of Dirichlet priors

Francesco Camaglia,Ilya Nemenman,Thierry Mora,Aleksandra M. Walczak

2023-07-09

Abstract:In many applications in biology, engineering and economics, identifying similarities and differences between distributions of data from complex processes requires comparing finite categorical samples of discrete counts. Statistical divergences quantify the difference between two distributions. However, their estimation is very difficult and empirical methods often fail, especially when the samples are small. We develop a Bayesian estimator of the Kullback-Leibler divergence between two probability distributions that makes use of a mixture of Dirichlet priors on the distributions being compared. We study the properties of the estimator on two examples: probabilities drawn from Dirichlet distributions, and random strings of letters drawn from Markov chains. We extend the approach to the squared Hellinger divergence. Both estimators outperform other estimation techniques, with better results for data with a large number of categories and for higher values of divergences.

Methodology,Information Theory,Data Analysis, Statistics and Probability
co-BPM: a Bayesian Model for Divergence Estimation

Kun Yang,Hao Su,Wing Hung Wong

DOI: https://doi.org/10.48550/arXiv.1410.0726

2014-10-02

Computation

Abstract:Divergence is not only an important mathematical concept in information theory, but also applied to machine learning problems such as low-dimensional embedding, manifold learning, clustering, classification, and anomaly detection. We proposed a bayesian model---co-BPM---to characterize the discrepancy of two sample sets, i.e., to estimate the divergence of their underlying distributions. In order to avoid the pitfalls of plug-in methods that estimate each density independently, our bayesian model attempts to learn a coupled binary partition of the sample space that best captures the landscapes of both distributions, then make direct inference on their divergences. The prior is constructed by leveraging the sequential buildup of the coupled binary partitions and the posterior is sampled via our specialized MCMC. Our model provides a unified way to estimate various types of divergences and enjoys convincing accuracy. We demonstrate its effectiveness through simulations, comparisons with the \emph{state-of-the-art} and a real data example.
The Logarithmic Super Divergence and its use in Statistical Inference

Avijit Maji,Abhik Ghosh,Ayanendranath Basu

DOI: https://doi.org/10.48550/arXiv.1407.3961

2014-07-15

Abstract:This paper introduces a new superfamily of divergences that is similar in spirit to the S-divergence family introduced by Ghosh et al. (2013). This new family serves as an umbrella that contains the logarithmic power divergence family (Renyi, 1961; Maji, Chakraborty and Basu 2014) and the logarithmic density power divergence family (Jones et al., 2001) as special cases. Various properties of this new family and the corresponding minimum distance procedures are discussed with particular emphasis on the robustness issue; these properties are demonstrated through simulation studies. In particular the method demonstrates the limitation of the first order influence function in assessing the robustness of the corresponding minimum distance procedures.

Methodology,Statistics Theory,Applications
Semi-Nonparametric Estimation of Distribution Divergence in Non-Euclidean Spaces

Chong Xiao Wang,Wee Peng Tay

2023-12-19

Abstract:This paper explores methods for estimating or approximating the total variation distance and the chi-squared divergence of probability measures within topological sample spaces, using independent and identically distributed samples. Our focus is on the practical scenario where the sample space is homeomorphic to subsets of Euclidean space, with the specific homeomorphism remaining unknown. Our proposed methods rely on the integral probability metric with witness functions in universal reproducing kernel Hilbert spaces (RKHSs). The estimators we develop consist of learnable parametric functions mapping the sample space to Euclidean space, paired with universal kernels defined in Euclidean space. This approach effectively overcomes the challenge of constructing universal kernels directly on non-Euclidean spaces. Furthermore, the estimators we devise demonstrate asymptotic consistency, and we provide a detailed statistical analysis, shedding light on their practical implementation.

Information Theory
t-divergence Based Approximate Inference

Ding, Nan,Qi, Yuan,Vishwanathan, S.v.n.

2011-01-01

Abstract:Approximate inference is an important technique for dealing with large, intractable graphical models based on the exponential family of distributions. We extend the idea of approximate inference to the t-exponential family by defining a new t-divergence. This divergence measure is obtained via convex duality between the log-partition function of the t-exponential family and a new t-entropy. We illustrate our approach on the Bayes Point Machine with a Student's t-prior.
The duo Fenchel-Young divergence

Frank Nielsen

DOI: https://doi.org/10.3390/e24030421

2022-03-17

Abstract:By calculating the Kullback-Leibler divergence between two probability measures belonging to different exponential families, we end up with a formula that generalizes the ordinary Fenchel-Young divergence. Inspired by this formula, we define the duo Fenchel-Young divergence and reports a majorization condition on its pair of generators which guarantees that this divergence is always non-negative. The duo Fenchel-Young divergence is also equivalent to a duo Bregman divergence. We show the use of these duo divergences by calculating the Kullback-Leibler divergence between densities of nested exponential families, and report a formula for the Kullback-Leibler divergence between truncated normal distributions. Finally, we prove that the skewed Bhattacharyya distance between nested exponential families amounts to an equivalent skewed duo Jensen divergence.

Information Theory
Limit Distribution Theory for Quantum Divergences

Sreejith Sreekumar,Mario Berta

2024-10-15

Abstract:Estimation of quantum relative entropy and its Rényi generalizations is a fundamental statistical task in quantum information theory, physics, and beyond. While several estimators of these divergences have been proposed in the literature along with their computational complexities explored, a limit distribution theory which characterizes the asymptotic fluctuations of the estimation error is still premature. As our main contribution, we characterize these asymptotic distributions in terms of Fréchet derivatives of elementary operator-valued functions. We achieve this by leveraging an operator version of Taylor's theorem and identifying the regularity conditions needed. As an application of our results, we consider an estimator of quantum relative entropy based on Pauli tomography of quantum states and show that the resulting asymptotic distribution is a centered normal, with its variance characterized in terms of the Pauli operators and states. We utilize the knowledge of the aforementioned limit distribution to obtain asymptotic performance guarantees for a multi-hypothesis testing problem.

Quantum Physics,Information Theory,Statistics Theory
Divergence Measure of Belief Function and Its Application in Data Fusion

Yutong Song,Yong Deng

DOI: https://doi.org/10.1109/access.2019.2932390

IF: 3.9

2019-01-01

IEEE Access

Abstract:Divergence measure is widely used in many applications. To efficiently deal with uncertainty in real applications, basic probability assignment ( $BPA$ ) in Dempster-Shafer evidence theory, instead of probability distribution, is adopted. As a result, an open issue is that how to measure the divergence of $BPA$ . In this paper, a new divergence measure of two $BPAtext{s}$ is proposed. The proposed divergence measure is the generalization of Kullback-Leibler divergence since when the $BPA$ is degenerated as probability distribution, the proposed belief divergence is equal to Kullback-Leibler divergence. Furthermore, compared with existing belief divergence measure, the new method has a better performance under the situation with a great degree of uncertainty and ambiguity. Numerical examples are used to illustrate the efficiency of the proposed divergence measure. In addition, based on the proposed belief divergence measure, a combination model is proposed to address data fusion. Finally, an example in target recognition is shown to illustrate the advantage of the new belief divergence in handling not only extreme uncertainty, but also highly conflicting data.

computer science, information systems,telecommunications,engineering, electrical & electronic
Unbiased Estimating Equation on Inverse Divergence and Its Conditions

Masahiro Kobayashi,Kazuho Watanabe

2024-04-25

Abstract:This paper focuses on the Bregman divergence defined by the reciprocal function, called the inverse divergence. For the loss function defined by the monotonically increasing function $f$ and inverse divergence, the conditions for the statistical model and function $f$ under which the estimating equation is unbiased are clarified. Specifically, we characterize two types of statistical models, an inverse Gaussian type and a mixture of generalized inverse Gaussian type distributions, to show that the conditions for the function $f$ are different for each model. We also define Bregman divergence as a linear sum over the dimensions of the inverse divergence and extend the results to the multi-dimensional case.

Information Theory,Machine Learning,Statistics Theory
Mutual Information Estimation via $f$-Divergence and Data Derangements

Nunzio A. Letizia,Nicola Novello,Andrea M. Tonello

2024-11-12

Abstract:Estimating mutual information accurately is pivotal across diverse applications, from machine learning to communications and biology, enabling us to gain insights into the inner mechanisms of complex systems. Yet, dealing with high-dimensional data presents a formidable challenge, due to its size and the presence of intricate relationships. Recently proposed neural methods employing variational lower bounds on the mutual information have gained prominence. However, these approaches suffer from either high bias or high variance, as the sample size and the structure of the loss function directly influence the training process. In this paper, we propose a novel class of discriminative mutual information estimators based on the variational representation of the $f$-divergence. We investigate the impact of the permutation function used to obtain the marginal training samples and present a novel architectural solution based on derangements. The proposed estimator is flexible since it exhibits an excellent bias/variance trade-off. The comparison with state-of-the-art neural estimators, through extensive experimentation within established reference scenarios, shows that our approach offers higher accuracy and lower complexity.

Machine Learning,Information Theory,Signal Processing
A Bayesian approach with Gaussian priors to the inverse problem of source identification in elliptic PDEs

Matteo Giordano

2024-07-25

Abstract:We consider the statistical linear inverse problem of making inference on an unknown source function in an elliptic partial differential equation from noisy observations of its solution. We employ nonparametric Bayesian procedures based on Gaussian priors, leading to convenient conjugate formulae for posterior inference. We review recent results providing theoretical guarantees on the quality of the resulting posterior-based estimation and uncertainty quantification, and we discuss the application of the theory to the important classes of Gaussian series priors defined on the Dirichlet-Laplacian eigenbasis and Matérn process priors. We provide an implementation of posterior inference for both classes of priors, and investigate its performance in a numerical simulation study.

Statistics Theory,Methodology
A Concentration Result of Estimating Phi-Divergence using Data Dependent Partition

Fengqiao Luo,Sanjay Mehrotra

DOI: https://doi.org/10.48550/arXiv.1801.00852

2018-01-03

Abstract:Estimation of the $\phi$-divergence between two unknown probability distributions using empirical data is a fundamental problem in information theory and statistical learning. We consider a multi-variate generalization of the data dependent partitioning method for estimating divergence between the two unknown distributions. Under the assumption that the distribution satisfies a power law of decay, we provide a convergence rate result for this method on the number of samples and hyper-rectangles required to ensure the estimation error is bounded by a given level with a given probability.

Probability,Statistics Theory
Bayesian influence diagnostics using normalizing functional Bregman divergence

Ian M Danilevicz,Ricardo S Ehlers

DOI: https://doi.org/10.48550/arXiv.1904.03717

2019-04-07

Applications

Abstract:Ideally, any statistical inference should be robust to local influences. Although there are simple ways to check about leverage points in independent and linear problems, more complex models require more sophisticated methods. Kullback-Leiber and Bregman divergences were already applied in Bayesian inference to measure the isolated impact of each observation in a model. We extend these ideas to models for dependent data and with non-normal probability distributions such as time series, spatial models and generalized linear models. We also propose a strategy to rescale the functional Bregman divergence to lie in the (0,1) interval thus facilitating interpretation and comparison. This is accomplished with a minimal computational effort and maintaining all theoretical properties. For computational efficiency, we take advantage of Hamiltonian Monte Carlo methods to draw samples from the posterior distribution of model parameters. The resulting Markov chains are then directly connected with Bregman calculus, which results in fast computation. We check the propositions in both simulated and empirical studies.
A new divergence measure for belief functions and its applications

Manpreet Kaur,Amit Srivastava

DOI: https://doi.org/10.1080/03081079.2022.2151006

2022-12-03

International Journal of General Systems

Abstract:Information fusion in uncertain and complex environments is highly challenging. Dempster–Shafer (D-S) evidence theory has been successfully applied by various researchers in multi-sensor data fusion. However, it yields counterintuitive results in case of highly conflicting evidence. In this paper, we have developed a new divergence measure for belief functions that is nonnegative, symmetric, and satisfies the triangle inequality. Using the developed divergence measure, an algorithm for combining distinct basic probability assignments (BPAs) has been discussed and applied in target recognition systems and classification problems.

computer science, theory & methods
On Semi-supervised Estimation of Discrete Distributions under f-divergences

Hasan Sabri Melihcan Erol,Lizhong Zheng

2024-05-16

Abstract:We study the problem of estimating the joint probability mass function (pmf) over two random variables. In particular, the estimation is based on the observation of $m$ samples containing both variables and $n$ samples missing one fixed variable. We adopt the minimax framework with $l^p_p$ loss functions. Recent work established that univariate minimax estimator combinations achieve minimax risk with the optimal first-order constant for $p \ge 2$ in the regime $m = o(n)$, questions remained for $p \le 2$ and various $f$-divergences. In our study, we affirm that these composite estimators are indeed minimax optimal for $l^p_p$ loss functions, specifically for the range $1 \le p \le 2$, including the critical $l_1$ loss. Additionally, we ascertain their optimality for a suite of $f$-divergences, such as KL, $\chi^2$, Squared Hellinger, and Le Cam divergences.

Statistics Theory,Information Theory
Towards a robust frequency-domain analysis: Spectral Rényi divergence revisited

Tetsuya Takabatake,Keisuke Yano

2023-10-11

Abstract:This paper studies a specific category of statistical divergences for spectral densities of time series: the spectral $\alpha$-Rényi divergences, which includes the Itakura-Saito divergence as a subset. While the spectral Rényi divergence has been acknowledged in past works, its statistical attributes have not been thoroughly investigated. The aim of this paper is to highlight these properties. We reveal a variational representation of spectral Rényi divergence, from which the minimum spectral Rényi divergence estimator is shown to be robust against outliers in the frequency domain, unlike the minimum Itakura-Saito divergence estimator, and thus it delivers more stable estimate, reducing the need for intricate pre-processing.

Statistics Theory
Bayesian Inference on Multivariate Medians and Quantiles

Indrabati Bhattacharya,Subhashis Ghosal

DOI: https://doi.org/10.5705/ss.202020.0108

2019-09-23

Abstract:In this paper, we consider Bayesian inference on a class of multivariate median and the multivariate quantile functionals of a joint distribution using a Dirichlet process prior. Since, unlike univariate quantiles, the exact posterior distribution of multivariate median and multivariate quantiles are not obtainable explicitly, we study these distributions asymptotically. We derive a Bernstein-von Mises theorem for the multivariate $\ell_1$-median with respect to general $\ell_p$-norm, which in particular shows that its posterior concentrates around its true value at $n^{-1/2}$-rate and its credible sets have asymptotically correct frequentist coverage. In particular, asymptotic normality results for the empirical multivariate median with general $\ell_p$-norm is also derived in the course of the proof which extends the results from the case $p=2$ in the literature to a general $p$. The technique involves approximating the posterior Dirichlet process by a Bayesian bootstrap process and deriving a conditional Donsker theorem. We also obtain analogous results for an affine equivariant version of the multivariate $\ell_1$-median based on an adaptive transformation and re-transformation technique. The results are extended to a joint distribution of multivariate quantiles. The accuracy of the asymptotic result is confirmed by a simulation study. We also use the results to obtain Bayesian credible regions for multivariate medians for Fisher's iris data, which consists of four features measured for each of three plant species.

Statistics Theory
Locally-Measured Rényi Divergences

Tobias Rippchen,Sreejith Sreekumar,Mario Berta

2024-05-08

Abstract:We propose an extension of the classical Rényi divergences to quantum states through an optimization over probability distributions induced by restricted sets of measurements. In particular, we define the notion of locally-measured Rényi divergences, where the set of allowed measurements originates from variants of locality constraints between (distant) parties $A$ and $B$. We then derive variational bounds on the locally-measured Rényi divergences and systematically discuss when these bounds become exact characterizations. As an application, we evaluate the locally-measured Rényi divergences on variants of highly symmetric data-hiding states, showcasing the reduced distinguishing power of locality-constrained measurements. For $n$-fold tensor powers, we further employ our variational formulae to derive corresponding additivity results, which gives the locally-measured Rényi divergences operational meaning as optimal rate exponents in asymptotic locally-measured hypothesis testing.

Quantum Physics
Divergence Inequalities with Applications in Ergodic Theory

Ian George,Alice Zheng,Akshay Bansal

2024-11-26

Abstract:The data processing inequality is central to information theory and motivates the study of monotonic divergences. However, it is not clear operationally we need to consider all such divergences. We establish a simple method for Pinsker inequalities as well as general bounds in terms of $\chi^{2}$-divergences for twice-differentiable $f$-divergences. These tools imply new relations for input-dependent contraction coefficients. We use these relations to show for many $f$-divergences the rate of contraction of a time homogeneous Markov chain is characterized by the input-dependent contraction coefficient of the $\chi^{2}$-divergence. This is efficient to compute and the fastest it could converge for a class of divergences. We show similar ideas hold for mixing times. Moreover, we extend these results to the Petz $f$-divergences in quantum information theory, albeit without any guarantee of efficient computation. These tools may have applications in other settings where iterative data processing is relevant.

Information Theory,Quantum Physics

On Bayesian estimation via divergences

Bayesian estimation of the Kullback-Leibler divergence for categorical systems using mixtures of Dirichlet priors

Bayesian estimation of the Kullback-Leibler divergence for categorical sytems using mixtures of Dirichlet priors

co-BPM: a Bayesian Model for Divergence Estimation

The Logarithmic Super Divergence and its use in Statistical Inference

Semi-Nonparametric Estimation of Distribution Divergence in Non-Euclidean Spaces

t-divergence Based Approximate Inference

The duo Fenchel-Young divergence

Limit Distribution Theory for Quantum Divergences

Divergence Measure of Belief Function and Its Application in Data Fusion

Unbiased Estimating Equation on Inverse Divergence and Its Conditions

Mutual Information Estimation via $f$-Divergence and Data Derangements

A Bayesian approach with Gaussian priors to the inverse problem of source identification in elliptic PDEs

A Concentration Result of Estimating Phi-Divergence using Data Dependent Partition

Bayesian influence diagnostics using normalizing functional Bregman divergence

A new divergence measure for belief functions and its applications

On Semi-supervised Estimation of Discrete Distributions under f-divergences

Towards a robust frequency-domain analysis: Spectral Rényi divergence revisited

Bayesian Inference on Multivariate Medians and Quantiles

Locally-Measured Rényi Divergences

Divergence Inequalities with Applications in Ergodic Theory