Discrete Stein characterizations and discrete information distances

Christophe Ley,Yvik Swan

DOI: https://doi.org/10.48550/arXiv.1201.0143

2011-12-30

Probability

Abstract:We construct two different Stein characterizations of discrete distributions and use these to provide a natural connection between Stein characterizations for discrete distributions and discrete information functionals.

What problem does this paper attempt to address?

Bounding the -Distance Between One-Dimensional Continuous and Discrete Distributions via Stein's Method

Gilles Germain,Yvik Swan

DOI: https://doi.org/10.1007/s10959-024-01373-x

2024-11-04

Journal of Theoretical Probability

Abstract:We introduce a new version of Stein's method of comparison of operators specifically tailored to the problem of bounding the (a.k.a. Wasserstein-1) distance between continuous and discrete distributions on the real line. Our approach rests on a new family of weighted discrete derivative operators, which we call bespoke derivatives. We also propose new bounds on the derivatives of the solutions of Stein equations for integrated Pearson random variables; this is a crucial step in Stein's method. We apply our result to several examples, including the central limit theorem, Pólya–Eggenberger urn models, the empirical distribution of the ground state of a many-interacting-worlds harmonic oscillator, the stationary distribution for the number of genes in the Moran model, and the stationary distribution of the Erlang-C system. Whenever our bounds can be compared with bounds from the literature, our constants are sharper.

statistics & probability
Stein's method for comparison of univariate distributions

Christophe Ley,Gesine Reinert,Yvik Swan

DOI: https://doi.org/10.48550/arXiv.1408.2998

2016-03-25

Abstract:We propose a new general version of Stein's method for univariate distributions. In particular we propose a canonical definition of the Stein operator of a probability distribution {which is based on a linear difference or differential-type operator}. The resulting Stein identity highlights the unifying theme behind the literature on Stein's method (both for continuous and discrete distributions). Viewing the Stein operator as an operator acting on pairs of functions, we provide an extensive toolkit for distributional comparisons. Several abstract approximation theorems are provided. Our approach is illustrated for comparison of several pairs of distributions : normal vs normal, sums of independent Rademacher vs normal, normal vs Student, and maximum of random variables vs exponential, Frechet and Gumbel.

Probability,Statistics Theory
One-dimensional Stein's method with bespoke derivatives

Gilles Germain,Yvik Swan

2023-11-03

Abstract:We introduce a version of Stein's method of comparison of operators specifically tailored to the problem of bounding the Wasserstein-1 distance between continuous and discrete distributions on the real line. Our approach rests on a new family of weighted discrete derivative operators, which we call bespoke derivatives. We also propose new bounds on the derivatives of the solutions of Stein equations for Integrated Pearson random variables; this is a crucial step in Stein's method. We apply our result to several examples, including the Central Limit Theorem, Polya-Eggenberger urn models, the empirical distribution of the ground state of a many-interacting-worlds harmonic oscillator, the stationary distribution for the number of genes in the Moran model, and the stationary distribution of the Erlang-C system. Whenever our bounds can be compared with bounds from the literature, our constants are sharper.

Probability
Information-Theoretic Measures on Lattices for High-Order Interactions

Zhaolu Liu,Mauricio Barahona,Robert L. Peach

2024-10-11

Abstract:Traditional models based solely on pairwise associations often fail to capture the complex statistical structure of multivariate data. Existing approaches for identifying information shared among groups of $d>3$ variables are frequently computationally intractable, asymmetric with respect to a target variable, or unable to account for all factorisations of the joint probability distribution. We present a systematic framework that derives higher-order information-theoretic measures using lattice and operator function pairs, whereby the lattice representing the algebraic relationships among variables, with operator functions that compute the measures over the lattice. We show that many commonly used measures can be derived within this framework, however they are often restricted to sublattices of the partition lattice, which prevents them from capturing all interactions when $d>3$. We also demonstrate that KL divergence, when used as an operator function, leads to unwanted cancellation of interactions for $d>3$. To fully characterise all interactions among $d$ variables, we introduce the Streitberg Information, using generalisations of KL divergence as an operator function, and defined over the full partition lattice. We validate Streitberg Information numerically on synthetic data, and illustrate its application in analysing complex interactions among stocks, decoding neural signals, and performing feature selection in machine learning.

Information Theory,Machine Learning
Distinction of the Steinberg representation with respect to a symmetric pair

Chuijia Wang,Jiandi Zou

2024-10-04

Abstract:Let $K$ be a non-archimedean local field of residual characteristic $p\neq 2$. Let $G$ be a connected reductive group over $K$, let $\theta$ be an involution of $G$ over $K$, and let $H$ be the connected component of $\theta$-fixed subgroup of $G$ over $K$. By realizing the Steinberg representation of $G$ as the $G$-space of complex smooth harmonic cochains following the idea of Broussous--Courtès, we study its space of distinction by $H$ as a finite dimensional complex vector space. We give an upper bound of the dimension, and under certain conditions, we show that the upper bound is sharp by explicitly constructing a basis using the technique of Poincaré series. Finally, we apply our general theory to the case where $G$ is a general linear group and $H$ a special orthogonal subgroup, which leads to a complete classification result.

Representation Theory,Number Theory
Stein's Method for Tempered Stable Distributions

Kalyan Barman,N. S. Upadhye

DOI: https://doi.org/10.48550/arXiv.2008.05818

2022-01-05

Abstract:In this article, we develop Stein characterization for two-sided tempered stable distribution. Stein characterizations for normal, gamma, Laplace, and variance-gamma distributions already known in the literature follow easily. One can also derive Stein characterizations for more difficult distributions such as the distribution of product of two normal random variables, a difference between two gamma random variables. Using the semigroup approach, we obtain estimates of the solution to Stein equation. Finally, we apply these estimates to obtain error bounds in the Wasserstein-type distance for tempered stable approximation in three well-known problems: comparison between two tempered stable distributions, Laplace approximation of random geometric sums, and six moment theorem for the symmetric variance-gamma approximation of functionals of double Wiener-It$\ddot{\text{o}}$ integrals. We also compare our results with the existing literature.

Probability
Stein Variational Inference for Discrete Distributions

Jun Han,Fan Ding,Xianglong Liu,Lorenzo Torresani,Jian Peng,Qiang Liu

DOI: https://doi.org/10.48550/arXiv.2003.00605

IF: 5.414

2020-03-01

Machine Learning

Abstract:Gradient-based approximate inference methods, such as Stein variational gradient descent (SVGD), provide simple and general-purpose inference engines for differentiable continuous distributions. However, existing forms of SVGD cannot be directly applied to discrete distributions. In this work, we fill this gap by proposing a simple yet general framework that transforms discrete distributions to equivalent piecewise continuous distributions, on which the gradient-free SVGD is applied to perform efficient approximate inference. The empirical results show that our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo on various challenging benchmarks of discrete graphical models. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN), outperforming other widely used ensemble methods on learning binarized AlexNet on CIFAR-10 dataset. In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions. Our proposed method outperforms existing GOF test methods for intractable discrete distributions.
On Stein Identity, Chernoff Inequality, and Orthogonal Polynomials

Zhengyuan Wei,Xinsheng Zhang,Taifu Li

DOI: https://doi.org/10.1080/03610920903068166

2010-01-01

Communication in Statistics- Theory and Methods

Abstract:In this article, we present a general method for deriving Stein-like identity and Chernoff-like inequality based on orthogonal polynomials. In order to illustrate our method, some applications are given with respect to normal, Gamma, Beta, Poisson, binomial, and negative binomial distribution, not only for random variables but also for random vectors, resulting corresponding Stein-like identity and Chernoff-like inequality are obtained consequently. Within our best knowledge, some of our matrix version results are new in the literature. In addition, forward difference formulae of Charlier polynomials, Krawtchouk polynomials and Meixner polynomials, Stein-like identity, and Chernoff-like inequality with respect to Beta distribution, as well as Rodrigues formula of Meixner polynomials are also prepared in the first time within our limited information. Interestingly, as far as normal, Gamma, Beta, Poisson, binomial, and negative binomial distribution are concerned, we found that their Stein-like identity and corresponding Chernoff-like inequality are related closely, by examining their Rodrigues formula.
A Stein characterisation of the distribution of the product of correlated normal random variables

Robert E. Gaunt,Siqi Li,Heather L. Sutcliffe

2024-09-10

Abstract:We obtain a Stein characterisation of the distribution of the product of two correlated normal random variables with non-zero means, and more generally the distribution of the sum of independent copies of such random variables. Our Stein characterisation is shown to naturally generalise a number of other Stein characterisations in the literature. From our Stein characterisation we derive recursive formulas for the moments of the product of two correlated normal random variables, and more generally the sum of independent copies of such random variables, which allows for efficient computation of higher order moments.

Probability
New Characterization of Blackwell's Order for Subsets of Information Structures

Danny Ben-Shahar,Eyal Sulganik

DOI: https://doi.org/10.1016/j.econlet.2023.111515

IF: 1.469

2024-01-05

Economics Letters

Abstract:We provide a new characterization of Blackwell's order for two subsets of stochastic information structures with meaningful economic interpretations. First, we show that one doubly stochastic information structure is strictly more informative than another, if and only if the convex hull of the former and all of its permutations, which are equally informative, contains that of the latter. We further present another subset of stochastic information structures for which we show that Blackwell's order is complete.

economics
On complementarity and distribution of imaginarity in finite dimensions

Bin Chen,Xiaofen Huang,Shao-Ming Fei

DOI: https://doi.org/10.1016/j.rinp.2024.107671

IF: 4.565

2024-04-14

Results in Physics

Abstract:We investigate the complementarity relation among the l1 norm of imaginarity, linear entropy and the Brukner–Zeilinger invariant information in the presence of conjugate symmetry. By using an equality derived from the complementarity relation, we define a quantity named index of imaginarity distribution with respect to the imaginarity, linear entropy and invariant information in finite dimensional quantum systems. We show that the index of imaginarity distribution is nonnegative and upper bounded by the square of the l1 norm of imaginarity, and serves as an invaluable tool in characterizing effectively the imaginarity distribution. Detailed examples are presented to illustrate our theoretical analysis.

physics, multidisciplinary,materials science
On the Steinness Index

Jihun Yum

DOI: https://doi.org/10.1007/s12220-018-0053-z

2018-06-19

Abstract:We introduce the concept of Steinness index related to the Stein neighborhood basis. We then show several results: (1) The existence of Steinness index is equivalent to that of strong Stein neighborhood basis. (2) On the Diederich–Fornæss worm domains in particular, we present an explicit formula relating the Steinness index to the well-known Diederich–Fornæss index. (3) The Steinness index is 1 if a smoothly bounded pseudoconvex domain admits finitely many boundary points of infinite type.
Reconstructing discrete measures from projections. Consequences on the empirical Sliced Wasserstein Distance

Eloi Tanguy,Rémi Flamary,Julie Delon

2024-04-12

Abstract:This paper deals with the reconstruction of a discrete measure $\gamma_Z$ on $\mathbb{R}^d$ from the knowledge of its pushforward measures $P_i\#\gamma_Z$ by linear applications $P_i: \mathbb{R}^d \rightarrow \mathbb{R}^{d_i}$ (for instance projections onto subspaces). The measure $\gamma_Z$ being fixed, assuming that the rows of the matrices $P_i$ are independent realizations of laws which do not give mass to hyperplanes, we show that if $\sum_i d_i > d$, this reconstruction problem has almost certainly a unique solution. This holds for any number of points in $\gamma_Z$. A direct consequence of this result is an almost-sure separability property on the empirical Sliced Wasserstein distance.

Mathematics
The Weil-Steinberg character of finite classical groups

G. Hiss,A. Zalesski

DOI: https://doi.org/10.48550/arXiv.0806.2567

2008-06-16

Abstract:We compute the irreducible constitutents of the product of the Weil character and the Steinberg character in those finite classical groups for which a Weil character is defined, namely the symplectic, unitary and general linear groups. It turns out that this product is multiplicity free for the symplectic and general unitary groups, but not for the general linear groups. As an application we show that the restriction of the Steinberg character of such a group to the subgroup stabilizing a vector in the natural module is multiplicity free. The proof of this result for the unitary groups uses an observation of Brunat, published as an appendix to our paper. As our "Weil character" for the symplectic groups in even characteristic we use the 2-modular Brauer character of the generalized spinor representation. Its product with the Steinberg character is the Brauer character of a projective module. We also determine its indecomposable direct summands.

Representation Theory,Group Theory
A Third Information-Theoretic Approach to Finite de Finetti Theorems

Mario Berta,Lampros Gavalakis,Ioannis Kontoyiannis

2024-04-26

Abstract:A new finite form of de Finetti's representation theorem is established using elementary information-theoretic tools. The distribution of the first $k$ random variables in an exchangeable vector of $n\geq k$ random variables is close to a mixture of product distributions. Closeness is measured in terms of the relative entropy and an explicit bound is provided. This bound is tighter than those obtained via earlier information-theoretic proofs, and its utility extends to random variables taking values in general spaces. The core argument employed has its origins in the quantum information-theoretic literature.

Information Theory,Probability,Quantum Physics
Understanding Higher-Order Interactions in Information Space

Herbert Edelsbrunner,Katharina Ölsböck,Hubert Wagner

DOI: https://doi.org/10.3390/e26080637

2024-07-27

Abstract:Methods used in topological data analysis naturally capture higher-order interactions in point cloud data embedded in a metric space. This methodology was recently extended to data living in an information space, by which we mean a space measured with an information theoretical distance. One such setting is a finite collection of discrete probability distributions embedded in the probability simplex measured with the relative entropy (Kullback-Leibler divergence). More generally, one can work with a Bregman divergence parameterized by a different notion of entropy. While theoretical algorithms exist for this setup, there is a paucity of implementations for exploring and comparing geometric-topological properties of various information spaces. The interest of this work is therefore twofold. First, we propose the first robust algorithms and software for geometric and topological data analysis in information space. Perhaps surprisingly, despite working with Bregman divergences, our design reuses robust libraries for the Euclidean case. Second, using the new software, we take the first steps towards understanding the geometric-topological structure of these spaces. In particular, we compare them with the more familiar spaces equipped with the Euclidean and Fisher metrics.
A de Bruijn identity for symmetric stable laws

Oliver Johnson

DOI: https://doi.org/10.48550/arXiv.1310.2045

2013-10-08

Abstract:We show how some attractive information--theoretic properties of Gaussians pass over to more general families of stable densities. We define a new score function for symmetric stable laws, and use it to give a stable version of the heat equation. Using this, we derive a version of the de Bruijn identity, allowing us to write the derivative of relative entropy as an inner product of score functions. We discuss maximum entropy properties of symmetric stable densities.

Information Theory,Probability
Discrete signature varieties

Carlo Bellingeri,Raul Penaguiao

2024-10-23

Abstract:Discrete signatures are invariants computed from time series corresponding to the discretised version of the signature of paths. We study the algebraic varieties arising from their images, the discrete signature varieties. We introduce them and compute their dimension in many cases. From a particular subclass of these varieties, we derive a partial solution to the Chen-Chow theorem for complex-valued time series.

Combinatorics,Algebraic Geometry,Probability
Stein characterizations for linear combinations of gamma random variables

Benjamin Arras,Ehsan Azmoodeh,Guillaume Poly,Yvik Swan

DOI: https://doi.org/10.1214/18-BJPS420

2017-09-06

Abstract:In this paper we propose a new, simple and explicit mechanism allowing to derive Stein operators for random variables whose characteristic function satisfies a simple ODE. We apply this to study random variables which can be represented as linear combinations of (non necessarily independent) gamma distributed random variables. The connection with Malliavin calculus for random variables in the second Wiener chaos is detailed. An application to McKay Type I random variables is also outlined.

Mathematics
Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem

Leon Lang,Pierre Baudot,Rick Quax,Patrick Forré

2024-09-23

Abstract:In information theory, one major goal is to find useful functions that summarize the amount of information contained in the interaction of several random variables. Specifically, one can ask how the classical Shannon entropy, mutual information, and higher interaction information relate to each other. This is answered by Hu's theorem, which is widely known in the form of information diagrams: it relates shapes in a Venn diagram to information functions, thus establishing a bridge from set theory to information theory. In this work, we view random variables together with the joint operation as a monoid that acts by conditioning on information functions, and entropy as a function satisfying the chain rule of information. This abstract viewpoint allows to prove a generalization of Hu's theorem. It applies to Shannon and Tsallis entropy, (Tsallis) Kullback-Leibler Divergence, cross-entropy, Kolmogorov complexity, submodular information functions, and the generalization error in machine learning. Our result implies for Chaitin's Kolmogorov complexity that the interaction complexities of all degrees are in expectation close to Shannon interaction information. For well-behaved probability distributions on increasing sequence lengths, this shows that the per-bit expected interaction complexity and information asymptotically coincide, thus showing a strong bridge between algorithmic and classical information theory.

Information Theory

Discrete Stein characterizations and discrete information distances

Bounding the -Distance Between One-Dimensional Continuous and Discrete Distributions via Stein's Method

Stein's method for comparison of univariate distributions

One-dimensional Stein's method with bespoke derivatives

Information-Theoretic Measures on Lattices for High-Order Interactions

Distinction of the Steinberg representation with respect to a symmetric pair

Stein's Method for Tempered Stable Distributions

Stein Variational Inference for Discrete Distributions

On Stein Identity, Chernoff Inequality, and Orthogonal Polynomials

A Stein characterisation of the distribution of the product of correlated normal random variables

New Characterization of Blackwell's Order for Subsets of Information Structures

On complementarity and distribution of imaginarity in finite dimensions

On the Steinness Index

Reconstructing discrete measures from projections. Consequences on the empirical Sliced Wasserstein Distance

The Weil-Steinberg character of finite classical groups

A Third Information-Theoretic Approach to Finite de Finetti Theorems

Understanding Higher-Order Interactions in Information Space

A de Bruijn identity for symmetric stable laws

Discrete signature varieties

Stein characterizations for linear combinations of gamma random variables

Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem