A note on the convergence of the Bayesian entropy estimator for exchangeable partitions

Servet Martinez

2023-11-18

Abstract:We show that when the proportions of a countable set of species are organized as an exchangeable partition of the unit interval and we take a sample on it, then the Bayesian posterior entropy converges a.s. and in L^1 to the entropy of the species when the sample size diverges to infinity.

Probability

What problem does this paper attempt to address?

Convergence of de Finetti's mixing measure in latent structure models for observed exchangeable sequences

Yun Wei,XuanLong Nguyen

DOI: https://doi.org/10.48550/arXiv.2004.05542

2021-09-24

Abstract:Mixtures of product distributions are a powerful device for learning about heterogeneity within data populations. In this class of latent structure models, de Finetti's mixing measure plays the central role for describing the uncertainty about the latent parameters representing heterogeneity. In this paper posterior contraction theorems for de Finetti's mixing measure arising from finite mixtures of product distributions will be established, under the setting the number of exchangeable sequences of observed variables increases while sequence length(s) may be either fixed or varied. The role of both the number of sequences and the sequence lengths will be carefully examined. In order to obtain concrete rates of convergence, a first-order identifiability theory for finite mixture models and a family of sharp inverse bounds for mixtures of product distributions will be developed via a harmonic analysis of such latent structure models. This theory is applicable to broad classes of probability kernels composing the mixture model of product distributions for both continuous and discrete domain $\mathfrak{X}$. Examples of interest include the case the probability kernel is only weakly identifiable in the sense of Ho and Nguyen (2016), the case where the kernel is itself a mixture distribution as in hierarchical models, and the case the kernel may not have a density with respect to a dominating measure on an abstract domain $\mathfrak{X}$ such as Dirichlet processes.

Statistics Theory,Probability
Entanglement and entropy in multipartite systems: a useful approach

A. Bernal,J. A. Casas,J.M. Moreno

DOI: https://doi.org/10.1007/s11128-023-04246-0

2024-02-15

Abstract:Quantum entanglement and quantum entropy are crucial concepts in the study of multipartite quantum systems. In this work we show how the notion of concurrence vector, re-expressed in a particularly useful form, provides new insights and computational tools for the analysis of both. In particular, using this approach for a general multipartite pure state, one can easily prove known relations in an easy way and to build up new relations between the concurrences associated with the different bipartitions. The approach is also useful to derive sufficient conditions for genuine entanglement in generic multipartite systems that are computable in polynomial time. From an entropy-of-entanglement perspective, the approach is powerful to prove properties of the Tsallis-$2$ entropy, such as the subadditivity, and to derive new ones, e.g. a modified version of the strong subadditivity which is always fulfilled; thanks to the purification theorem these results hold for any multipartite state, whether pure or mixed.

Quantum Physics,High Energy Physics - Theory
When you can't count, sample! Computable entropies beyond equilibrium from basin volumes

Mathias Casiulis,Stefano Martiniani

DOI: https://doi.org/10.4279/pip.150001

2023-12-13

Abstract:In statistical mechanics, measuring the number of available states and their probabilities, and thus the system's entropy, enables the prediction of the macroscopic properties of a physical system at equilibrium. This predictive capacity hinges on the knowledge of the a priori probabilities of observing the states of the system, given by the Boltzmann distribution. Unfortunately, the successes of equilibrium statistical mechanics are hard to replicate out of equilibrium, where the a priori probabilities of observing states are in general not known, precluding the naïve application of usual tools. In the last decade, exciting developments have occurred that enable the direct numerical estimation of the entropy and density of states of athermal and non-equilibrium systems, thanks to significant methodological advances in the computation of the volume of high-dimensional basins of attraction. Here, we provide a detailed account of these methods, underscoring the challenges that lie in such estimations, recent progress on the matter, and promising directions for future work.

Statistical Mechanics,Computational Physics
Conditional Entropy Formula with Respect to Monotonic Partitions

Weisheng Wu,Yiwei Zhang,Xiaomin Zhou

DOI: https://doi.org/10.1007/s10883-024-09682-4

IF: 1.27

2024-02-22

Journal of Dynamical and Control Systems

Abstract:We investigate conditional entropy with respect to monotonic (invariant, decreasing, or increasing) measurable partitions. In particular, we obtain Brin-Katok's and Katok's entropy formula for conditional entropy with respect to invariant, decreasing and a large class of increasing partitions. Our proofs develop new adaptions on applying conditional variants of Shannon-McMillan-Breiman theorems corresponding to the monotonic partitions, to link with the local and global conditional entropy forms. As an application, we obtain a number of reformulations of the unstable entropy for ergodic measures of general diffeomorphisms.

mathematics, applied,automation & control systems
Statistical inference with exchangeability and martingales

Chris C. Holmes,Stephen G. Walker

DOI: https://doi.org/10.1098/rsta.2022.0143

2023-03-29

Abstract:In this paper, we start by reviewing exchangeability and its relevance to the Bayesian approach. We highlight the predictive nature of Bayesian models and the symmetry assumptions implied by beliefs of an underlying exchangeable sequence of observations. By taking a closer look at the Bayesian bootstrap, the parametric bootstrap of Efron and a version of Bayesian thinking about inference uncovered by Doob based on martingales, we introduce a parametric Bayesian bootstrap. Martingales play a fundamental role. Illustrations are presented as is the relevant theory. This article is part of the theme issue ‘Bayesian inference: challenges, perspectives, and prospects’.

multidisciplinary sciences
Inference of Extreme Synchrony with an Entropy Measure on a Bipartite Network

Aki-Hiro Sato

DOI: https://doi.org/10.1109/compsac.2017.150

2017-07-01

Abstract:This article proposes a method to quantify the structure of a bipartite graph using a network entropy per link. Collective behavior of participants' quotes and trades in the foreign exchange market is quantified by applying the proposed method to high frequency data. The network entropy per link corresponded to the macroeconomic situation. A finite mixture of Gumbel distributions is used to fit the empirical distribution for the minimum values of network entropy per link in each week. The mixture of Gumbel distributions with parameter estimates by segmentation procedure was verified by the Kolmogorov-Smirnov test. The finite mixture of Gum-bel distributions that extrapolate the empirical probability of extreme events has explanatory power at a statistically significant level. These findings provide insight into how to manage collective behavior in the foreign exchange market.
Finite-sample concentration of the empirical relative entropy around its mean

Rohit Agrawal

DOI: https://doi.org/10.48550/arXiv.2203.00800

2022-03-02

Abstract:In this note, we show that the relative entropy of an empirical distribution of $n$ samples drawn from a set of size $k$ with respect to the true underlying distribution is exponentially concentrated around its expectation, with central moment generating function bounded by that of a gamma distribution with shape $2k$ and rate $n/2$. This improves on recent work of Bhatt and Pensia (arXiv 2021) on the same problem, who showed such a similar bound with an additional polylogarithmic factor of $k$ in the shape, and also confirms a recent conjecture of Mardia et al. (Information and Inference 2020). The proof proceeds by reducing the case $k>3$ of the multinomial distribution to the simpler case $k=2$ of the binomial, for which the desired bound follows from standard results on the concentration of the binomial.

Statistics Theory,Information Theory,Probability
Better bounds for the union-closed sets conjecture using the entropy approach

Stijn Cambie

DOI: https://doi.org/10.48550/arXiv.2212.12500

2022-12-23

Combinatorics

Abstract:We improve the best known constant $\frac{3-\sqrt 5}{2}$ for which the union-closed conjecture is known to be true, by using dependent samples as suggested by Sawin and the entropy approach on this problem initiated by Gilmer. Meanwhile, we focus on the intuition behind this entropy approach and its boundaries.
A note on maximal conditional entropy on Lebesgue spaces

Michael Hediger

2024-04-22

Abstract:Let $(X,\mathcal{B},P)$ be a probability space and $\mathit{a}$ be a sub $\sigma$-field that is generated by an increasing sequence of sub $\sigma$-fields $(\mathit{a}_{n})_{n \in \mathbb{N}}$. Given $\theta \in \Theta$, where $\Theta$ is some set, let $(X_{n}^{\theta})_{n \in \mathbb{N}}$ be a martingale adapted to $(\mathit{a}_{n})_{n \in \mathbb{N}}$. Martin (1969) provides sufficient conditions to show that $(X_{n}^{\theta})_{n \in \mathbb{N}}$ converges a.s. uniformly on $\Theta$ to a random variable $X^{\theta}$. His results are based on the assumption that there exists an integer $n$ s.t. the conditional entropy given $\mathit{a}_{n}$ is uniformly bounded over the set of finite partitions of $X$ with atoms from $\mathit{a}$. This study complements Martin's results by studying the latter assumption on the maximal conditional entropy in the context of measurable partitions of Lebesgue spaces. We provide conditions under which $\mathit{a}$ conveys too much information for the maximal conditional entropy to be finite. As an example, we consider the space of continuous functions with a compact support, equipped with the Borel $\sigma$-field.

Probability
Estimation of Tsallis entropy for exponentially distributed several populations

Naveen Kumar,Ambesh Dixit,Vivek Vijay

2024-01-17

Abstract:We study the estimation of Tsallis entropy of a finite number of independent populations, each following an exponential distribution with the same scale parameter and distinct location parameters for $q>0$. We derive a Stein-type improved estimate, establishing the inadmissibility of the best affine equivariant estimate of the parameter function. A class of smooth estimates utilizing the Brewster technique is obtained, resulting in a significant improvement in the risk value. We computed the Brewster-Zidek estimates for both one and two populations, to illustrate the comparison with best affine equivariant and Stein-type estimates. We further derive that the Bayesian estimate, employing an inverse gamma prior, which takes the best affine equivariant estimate as a particular case. We provide a numerical illustration utilizing simulated samples for a single population. The purpose is to demonstrate the impact of sample size, location parameter, and entropic index on the estimates.

Statistics Theory
Convergence rates of empirical Bayesian estimation in a class of linear models

Laisheng Wei

IF: 1.4

1998-01-01

Statistica Sinica

Abstract:By using kernel estimators of a multivariate density function and its partial derivatives, and the estimators of the nuisance parameters, we construct empirical Bayes (EB) estimators of parameters in a class of linear models. Under suitable moment conditions on the prior distribution, the proposed EB estimators are asymptotically optimal with rates arbitrarily close to O(n(-1)).
A Note on the Convergence Rates for Empirical Bayes Estimators of Parameters in Multiple-Parameter Exponential Families

SP Zhang,LS Wei

DOI: https://doi.org/10.1080/03610929908832357

1999-01-01

Abstract:Under suitable conditions upon prior distribution, the convergence rates for empirical Bayes estimators of parameters in multi-parameter exponential families (M-PEF) are obtained. It is shown that the assumptions Tong (1996) imposed on the marginal density can be reduced. The above result can also be extended to more general forms of M-PEF. Finally, some examples which satisfy the conditions of the theorems are given.
Fast Generation of Exchangeable Sequence of Clusters Data

Keith Levin,Brenda Betancourt

DOI: https://doi.org/10.48550/arXiv.2209.02844

2022-09-07

Abstract:Recent advances in Bayesian models for random partitions have led to the formulation and exploration of Exchangeable Sequences of Clusters (ESC) models. Under ESC models, it is the cluster sizes that are exchangeable, rather than the observations themselves. This property is particularly useful for obtaining microclustering behavior, whereby cluster sizes grow sublinearly in the number of observations, as is common in applications such as record linkage, sparse networks and genomics. Unfortunately, the exchangeable clusters property comes at the cost of projectivity. As a consequence, in contrast to more traditional Dirichlet Process or Pitman-Yor process mixture models, samples a priori from ESC models cannot be easily obtained in a sequential fashion and instead require the use of rejection or importance sampling. In this work, drawing on connections between ESC models and discrete renewal theory, we obtain closed-form expressions for certain ESC models and develop faster methods for generating samples a priori from these models compared with the existing state of the art. In the process, we establish analytical expressions for the distribution of the number of clusters under ESC models, which was unknown prior to this work.

Statistics Theory
Large deviations of return times and related entropy estimators on shift spaces

Noé Cuneo,Renaud Raquépas

2024-05-22

Abstract:We prove the large deviation principle for several entropy and cross entropy estimators based on return times and waiting times on shift spaces over finite alphabets. We consider shift-invariant probability measures satisfying some decoupling conditions which imply no form of mixing nor ergodicity. We establish precise relations between the rate functions of the different estimators, and between these rate functions and the corresponding pressures, one of which is the Rényi entropy function. For the most commonly used definition of return times, the large-deviation rate function is proved to be nonconvex, except in marginal cases. The results apply in particular to irreducible Markov chains, equilibrium measures for Bowen-regular potentials, g-measures, invariant Gibbs states for absolutely summable interactions in statistical mechanics, and also to probability measures which may be far from Gibbsian, including some hidden Markov models and repeated quantum measurement processes.

Probability,Mathematical Physics,Dynamical Systems
Fast generation of exchangeable sequences of clusters data

Keith Levin,Brenda Betancourt

DOI: https://doi.org/10.1007/s11222-024-10385-w

IF: 2.3241

2024-02-09

Statistics and Computing

Abstract:Recent advances in Bayesian models for random partitions have led to the formulation and exploration of Exchangeable Sequences of Clusters (ESC) models. Under ESC models, it is the cluster sizes that are exchangeable, rather than the observations themselves. This property is particularly useful for obtaining microclustering behavior, whereby cluster sizes grow sublinearly in the number of observations, as is common in applications such as record linkage, sparse networks and genomics. Unfortunately, the exchangeable clusters property comes at the cost of projectivity. As a consequence, in contrast to more traditional Dirichlet Process or Pitman–Yor process mixture models, samples a priori from ESC models cannot be easily obtained in a sequential fashion and instead require the use of rejection or importance sampling. In this work, drawing on connections between ESC models and discrete renewal theory, we obtain closed-form expressions for certain ESC models and develop faster methods for generating samples a priori from these models compared with the existing state of the art. In the process, we establish analytical expressions for the distribution of the number of clusters under ESC models, which was unknown prior to this work.

statistics & probability,computer science, theory & methods
Robust random walk-like Metropolis-Hastings algorithms for concentrating posteriors

Daniel Rudolf,Björn Sprungk

DOI: https://doi.org/10.48550/arXiv.2202.12127

2022-02-24

Abstract:Motivated by Bayesian inference with highly informative data we analyze the performance of random walk-like Metropolis-Hastings algorithms for approximate sampling of increasingly concentrating target distributions. We focus on Gaussian proposals which use a Hessian-based approximation of the target covariance. By means of pushforward transition kernels we show that for Gaussian target measures the spectral gap of the corresponding Metropolis-Hastings algorithm is independent of the concentration of the posterior, i.e., the noise level in the observational data that is used for Bayesian inference. Moreover, by exploiting the convergence of the concentrating posteriors to their Laplace approximation we extend the analysis to non-Gaussian target measures which either concentrate around a single point or along a linear manifold. In particular, in that setting we show that the average acceptance rate as well as the expected squared jump distance of suitable Metropolis-Hastings Markov chains do not deteriorate as the target concentrates.

Computation,Numerical Analysis,Probability,Statistics Theory
Separate Exchangeability as Modeling Principle in Bayesian Nonparametrics

Giovanni Rebaudo,Qiaohui Lin,Peter Mueller

2024-06-20

Abstract:We argue for the use of separate exchangeability as a modeling principle in Bayesian nonparametric (BNP) inference. Separate exchangeability is \emph{de facto} widely applied in the Bayesian parametric case, e.g., it naturally arises in simple mixed models. However, while in some areas, such as random graphs, separate and (closely related) joint exchangeability are widely used, it is curiously underused for several other applications in BNP. We briefly review the definition of separate exchangeability focusing on the implications of such a definition in Bayesian modeling. We then discuss two tractable classes of models that implement separate exchangeability that are the natural counterparts of familiar partially exchangeable BNP models. The first is nested random partitions for a data matrix, defining a partition of columns and nested partitions of rows, nested within column clusters. Many recent models for nested partitions implement partially exchangeable models related to variations of the well-known nested Dirichlet process. We argue that inference under such models in some cases ignores important features of the experimental setup. We obtain the separately exchangeable counterpart of such partially exchangeable partition structures. The second class is about setting up separately exchangeable priors for a nonparametric regression model when multiple sets of experimental units are involved. We highlight how a Dirichlet process mixture of linear models known as ANOVA DDP can naturally implement separate exchangeability in such regression problems. Finally, we illustrate how to perform inference under such models in two real data examples.

Methodology,Statistics Theory
Error Bounds on a Mixed Entropy Inequality

James Melbourne,Saurav Talukdar,Shreyas Bhaban,Murti V. Salapaka

DOI: https://doi.org/10.1109/isit.2018.8437601

2018-06-01

Abstract:Motivated by the entropy computations relevant to the evaluation of decrease in entropy in bit reset operations, the authors investigate the deficit in an entropic inequality involving two independent random variables, one continuous and the other discrete. In the case where the continuous random variable is Gaussian, we derive strong quantitative bounds on the deficit in the inequality. More explicitly it is shown that the decay of the deficit is sub-Gaussian with respect to the reciprocal of the standard deviation of the Gaussian variable. What is more, up to rational terms these results are shown to be sharp.
Growing integer partitions with uniform marginals and the equivalence of partition ensembles

Yuri Yakubovich

DOI: https://doi.org/10.1016/j.aim.2024.109908

2024-09-28

Abstract:We present an explicit construction of a Markovian random growth process on integer partitions such that given it visits some level $n$, it passes through any partition $\lambda$ of $n$ with equal probabilities. The construction has continuous time, but we also investigate its discrete time jump chain. The jump probabilities are given by explicit but complicated expressions, so we find their asymptotic behavior as the partition becomes large. This allows us to explain how the limit shape is formed. Using the known connection of the considered probabilistic objects to Poisson point processes, we give an alternative description of the partition growth process in these terms. Then we apply the constructed growth process to find sufficient conditions for a phenomenon known as equivalence of two ensembles of random partitions for a finite number of partition characteristics. This result allows to show that counts of odd and even parts in a random partition of $n$ are asymptotically independent as $n\to\infty$ and to find their limiting distributions, which are, somewhat surprisingly, different.

Probability

A note on the convergence of the Bayesian entropy estimator for exchangeable partitions

Convergence of de Finetti's mixing measure in latent structure models for observed exchangeable sequences

Entanglement and entropy in multipartite systems: a useful approach

When you can't count, sample! Computable entropies beyond equilibrium from basin volumes

Conditional Entropy Formula with Respect to Monotonic Partitions

Statistical inference with exchangeability and martingales

Inference of Extreme Synchrony with an Entropy Measure on a Bipartite Network

Finite-sample concentration of the empirical relative entropy around its mean

Better bounds for the union-closed sets conjecture using the entropy approach

A note on maximal conditional entropy on Lebesgue spaces

Estimation of Tsallis entropy for exponentially distributed several populations

Convergence rates of empirical Bayesian estimation in a class of linear models

A Note on the Convergence Rates for Empirical Bayes Estimators of Parameters in Multiple-Parameter Exponential Families

Fast Generation of Exchangeable Sequence of Clusters Data

Large deviations of return times and related entropy estimators on shift spaces

Fast generation of exchangeable sequences of clusters data

Robust random walk-like Metropolis-Hastings algorithms for concentrating posteriors

Separate Exchangeability as Modeling Principle in Bayesian Nonparametrics

Error Bounds on a Mixed Entropy Inequality

Growing integer partitions with uniform marginals and the equivalence of partition ensembles