When the Cramer-Rao Inequality provides no information

Steven J. Miller

DOI: https://doi.org/10.48550/arXiv.0704.0923

2007-04-07

Abstract:We investigate a one-parameter family of probability densities (related to the Pareto distribution, which describes many natural phenomena) where the Cramer-Rao inequality provides no information.

Statistics Theory

What problem does this paper attempt to address?

Some families of Jensen-like inequalities with application to information theory

Neri Merhav

DOI: https://doi.org/10.3390/e25050752

2023-04-06

Abstract:It is well known that the traditional Jensen inequality is proved by lower bounding the given convex function, $f(x)$, by the tangential affine function that passes through the point $(E\{X\},f(E\{X\}))$, where $E\{X\}$ is the expectation of the random variable $X$. While this tangential affine function yields the tightest lower bound among all lower bounds induced by affine functions that are tangential to $f$, it turns out that when the function $f$ is just part of a more complicated expression whose expectation is to be bounded, the tightest lower bound might belong to a tangential affine function that passes through a point different than $(E\{X\},f(E\{X\}))$. In this paper, we take advantage of this observation, by optimizing the point of tangency with regard to the specific given expression, in a variety of cases, and thereby derive several families of inequalities, henceforth referred to as ``Jensen-like'' inequalities, which are new to the best knowledge of the author. The degree of tightness and the potential usefulness of these inequalities is demonstrated in several application examples related to information theory.

Information Theory
The Cramér‐Rao Lower Bound on Variance: Adam and Eve's “uncertainty Principle”

Michael R. Powers

DOI: https://doi.org/10.1108/15265940610681637

2006-01-01

Abstract:Purpose - The editorial aims to describe the importance of the Cramer-Rao lower bound (CRLB) in matters of risk, and consider why this inequality generally receives less attention than it deserves.Design/methodology/approach - The CRLB is discussed in the context of other celebrated equations, one of which (Heisenberg's uncertainty principle) is actually implied by the CRLB. A hypothetical world with no randomness on the macroscopic level is then imagined, and the role of the CRLB in that world is considered.Findings - It is found that in a world with no macroscopic uncertainty, the CRLB would play a central role in describing the "counterintuitive" concept of randomness.Originality/value - The editorial identifies and explores an important epistemological result that is under-appreciated.
A constrained risk inequality for general losses

John C. Duchi,Feng Ruan

2018-04-22

Abstract:We provide a general constrained risk inequality that applies to arbitrary non-decreasing losses, extending a result of Brown and Low [Ann. Stat. 1996]. Given two distributions $P_0$ and $P_1$, we find a lower bound for the risk of estimating a parameter $\theta(P_1)$ under $P_1$ given an upper bound on the risk of estimating the parameter $\theta(P_0)$ under $P_0$. The inequality is a useful pedagogical tool, as its proof relies only on the Cauchy-Schwartz inequality, it applies to general losses, and it transparently gives risk lower bounds on super-efficient and adaptive estimators.

Statistics Theory
A Matrix Bernstein Inequality for Strong Rayleigh Distributions

Tarun Kathuria

DOI: https://doi.org/10.48550/arXiv.2011.13340

2020-11-26

Abstract:The Entropy method provides a powerful framework for proving scalar concentration inequalities by establishing functional inequalities like Poincare and log-Sobolev inequalities. These inequalities are especially useful for deriving concentration for dependent distributions coming from stationary distributions of Markov chains. In contrast to the scalar case, the study of matrix concentration inequalities is fairly recent and there does not seem to be a general framework for proving matrix concentration inequalities using the Entropy method. In this paper, we initiate the study of Matrix valued functional inequalities and show how matrix valued functional inequalities can be used to establish a kind of iterative variance comparison argument to prove Matrix Bernstein like inequalities. Strong Rayleigh distributions are a class of distributions which satisfy a strong form of negative dependence properties. We show that scalar Poincare inequalities recently established for a flip-swap markov chain considered in the work of Hermon and Salez [HS19] can be lifted to prove matrix valued Poincare inequalities. This can then be combined with the iterative variance argument to prove a version of matrix Bernstein inequality for Lipshitz matrix-valued functions for Strong Rayleigh measures.

Probability
A Note on the Harnack Inequality Related with the Martin Boundary

H. -J. Ruan,X. -Y. Wang

2015-01-01

Markov processes and related fields

Abstract:Harnack inequality plays a key role for the homeomorphism between the Martin boundaries and the self-similar sets. In this note, we study a transition probability on the finite words space of an iterated function system, whose invariant set is the unit interval. We present the necessary and sufficient condition for the transition probability to satisfy the Harnack inequality.
Information-Theoretic Perspectives on Brascamp-Lieb Inequality and Its Reverse

Jingbo Liu,Thomas A. Courtade,Paul Cuff,Sergio Verdu

DOI: https://doi.org/10.48550/arXiv.1702.06260

2017-12-03

Abstract:We introduce an inequality which may be viewed as a generalization of both the Brascamp-Lieb inequality and its reverse (Barthe's inequality), and prove its information-theoretic (i.e.\ entropic) formulation. This result leads to a unified approach to functional inequalities such as the variational formula of Rényi entropy, hypercontractivity and its reverse, strong data processing inequalities, and transportation-cost inequalities, whose utility in the proofs of various coding theorems has gained growing popularity recently. We show that our information-theoretic setting is convenient for proving properties such as data processing, tensorization, convexity (Riesz-Thorin interpolation) and Gaussian optimality. In particular, we elaborate on a "doubling trick" used by Lieb and Geng-Nair to prove several results on Gaussian optimality. Several applications are discussed, including a generalization of the Brascamp-Lieb inequality involving Gaussian random transformations, the determination of Wyner's common information of vector Gaussian sources, and the achievable rate region of certain key generation problems in the case of vector Gaussian sources.

Information Theory
Information inequality problem over set functions

Miika Hannula

2023-09-21

Abstract:Information inequalities appear in many database applications such as query output size bounds, query containment, and implication between data dependencies. Recently Khamis et al. proposed to study the algorithmic aspects of information inequalities, including the information inequality problem: decide whether a linear inequality over entropies of random variables is valid. While the decidability of this problem is a major open question, applications often involve only inequalities that adhere to specific syntactic forms linked to useful semantic invariance properties. This paper studies the information inequality problem in different syntactic and semantic scenarios that arise from database applications. Focusing on the boundary between tractability and intractability, we show that the information inequality problem is coNP-complete if restricted to normal polymatroids, and in polynomial time if relaxed to monotone functions. We also examine syntactic restrictions related to query output size bounds, and provide an alternative proof, through monotone functions, for the polynomial-time computability of the entropic bound over simple sets of degree constraints.

Databases,Information Theory
The One Dimensional Free Poincaré Inequality

Michel Ledoux,Ionel Popescu

DOI: https://doi.org/10.48550/arXiv.1105.2031

2012-04-21

Abstract:In this paper we discuss the natural candidate for the one dimensional free Poincaré inequality. Two main strong points sustain this candidacy. One is the random matrix heuristic and the other the relations with the other free functional inequalities, namely, the free transportation and Log-Sobolev inequalities. As in the classical case the Poincaré is implied by the others. This investigation is driven by a nice lemma of Haagerup which relates logarithmic potentials and Chebyshev polynomials. The Poincaré inequality revolves around the counting number operator for the Chebyshev polynomials of first kind with respect to the arcsine law on $[-2,2]$. This counting number operator appears naturally in a representation of the minimum of the logarithmic potential with external fields as well as in the perturbation of logarithmic energy with external fields, which is the essential connection between all these inequalities.

Operator Algebras,Functional Analysis
Brascamp-Lieb Inequality and Its Reverse: An Information Theoretic View

Jingbo Liu,Thomas A. Courtade,Paul Cuff,Sergio Verdu

DOI: https://doi.org/10.48550/arXiv.1605.02818

2016-05-10

Abstract:We generalize a result by Carlen and Cordero-Erausquin on the equivalence between the Brascamp-Lieb inequality and the subadditivity of relative entropy by allowing for random transformations (a broadcast channel). This leads to a unified perspective on several functional inequalities that have been gaining popularity in the context of proving impossibility results. We demonstrate that the information theoretic dual of the Brascamp-Lieb inequality is a convenient setting for proving properties such as data processing, tensorization, convexity and Gaussian optimality. Consequences of the latter include an extension of the Brascamp-Lieb inequality allowing for Gaussian random transformations, the determination of the multivariate Wyner common information for Gaussian sources, and a multivariate version of Nelson's hypercontractivity theorem. Finally we present an information theoretic characterization of a reverse Brascamp-Lieb inequality involving a random transformation (a multiple access channel).

Information Theory
Transportation-information Inequalities for Markov Processes

Arnaud Guillin,Christian Léonard,Liming Wu,Nian Yao

DOI: https://doi.org/10.1007/s00440-008-0159-5

IF: 1.944

2008-01-01

Probability Theory and Related Fields

Abstract:In this paper, one investigates the following type of transportation-information $T_cI$ inequalities: $\alpha(T_c(\nu,\mu))\le I(\nu|\mu)$ for all probability measures $\nu$ on some metric space $(\XX, d)$, where $\mu$ is a given probability measure, $T_c(\nu,\mu)$ is the transportation cost from $\nu$ to $\mu$ with respect to some cost function $c(x,y)$ on $\XX^2$, $I(\nu|\mu)$ is the Fisher-Donsker-Varadhan information of $\nu$ with respect to $\mu$ and $\alpha: [0,\infty)\to [0,\infty]$ is some left continuous increasing function. Using large deviation techniques, it is shown that $T_cI$ is equivalent to some concentration inequality for the occupation measure of a $\mu$-reversible ergodic Markov process related to $I(\cdot|\mu)$, a counterpart of the characterizations of transportation-entropy inequalities, recently obtained by Gozlan and L\'eonard in the i.i.d. case . Tensorization properties of $T_cI$ are also derived.
Matrix Infinitely Divisible Series: Tail Inequalities and Their Applications

Chao Zhang,Xianjie Gao,Min-Hsiu Hsieh,Hanyuan Hang,Dacheng Tao

DOI: https://doi.org/10.1109/tit.2019.2951759

IF: 2.5

2019-01-01

IEEE Transactions on Information Theory

Abstract:In this paper, we study tail inequalities of the largest eigenvalue of a matrix infinitely divisible (i.d.) series, which is a finite sum of fixed matrices weighted by i.d. random variables. We obtain several types of tail inequalities, including Bennett-type and Bernstein-type inequalities. This allows us to further bound the expectation of the spectral norm of a matrix i.d. series. Moreover, by developing a new lower-bound function for $Q(s)=(s+1)\log (s+1)-s$ that appears in the Bennett-type inequality, we derive a tighter tail inequality of the largest eigenvalue of the matrix i.d. series than the Bernstein-type inequality when the matrix dimension is high. The resulting lower-bound function is of independent interest and can improve any Bennett-type concentration inequality that involves the function $Q(s)$ . The class of i.d. probability distributions is large and includes Gaussian and Poisson distributions, among many others. Therefore, our results encompass the existing work on matrix Gaussian series as a special case. Lastly, we show that the tail inequalities of a matrix i.d. series have applications in several optimization problems including the chance constrained optimization problem and the quadratic optimization problem with orthogonality constraints. In addition, we also use the resulting tail bounds to show that random matrices constructed from i.d. random variables satisfy the restricted isometry property (RIP) when it acts as a measurement matrix in compressed sensing.
A Matrix Version of Chernoff Inequality

Zhengyuan Wei,Xinsheng Zhang

DOI: https://doi.org/10.1016/j.spl.2008.01.044

IF: 0.718

2008-01-01

Statistics & Probability Letters

Abstract:An interesting result from the point of view of upper variance bounds is the inequality of Chernoff [Chernoff, H., 1981. A note on an inequality involving the normal distribution. Annals of Probability 9, 533–535]. Namely, that for every absolutely continuous function g with derivative g′ such that Var{|g(ξ)|}<∞, and for standard normal r.v. ξ, Var(|g(ξ)|)≤E{(g′(ξ))2}. Both the usefulness and simplicity of this inequality have generated a great deal of extensions, as well as alternative proofs. Particularly, Olkin and Shepp [Olkin, I., Shepp, L., 2005. A matrix variance inequality. Journal of Statistical Planning and Inference 130, 351–358] obtained an inequality for the covariance matrix of k functions. However, all the previous papers have focused on univariate function and univariate random variable. We provide here a covariance matrix inequality for multivariate function of multivariate normal distribution.
Bounds on Portfolio Quality

Steven E. Pav

DOI: https://doi.org/10.48550/arXiv.1409.5936

2014-09-21

Abstract:The signal-noise ratio of a portfolio of p assets, its expected return divided by its risk, is couched as an estimation problem on the sphere. When the portfolio is built using noisy data, the expected value of the signal-noise ratio is bounded from above via a Cramer-Rao bound, for the case of Gaussian returns. The bound holds for `biased' estimators, thus there appears to be no bias-variance tradeoff for the problem of maximizing the signal-noise ratio. An approximate distribution of the signal-noise ratio for the Markowitz portfolio is given, and shown to be fairly accurate via Monte Carlo simulations, for Gaussian returns as well as more exotic returns distributions. These findings imply that if the maximal population signal-noise ratio grows slower than the universe size to the 1/4 power, there may be no diversification benefit, rather expected signal-noise ratio can decrease with additional assets. As a practical matter, this may explain why the Markowitz portfolio is typically applied to small asset universes. Finally, the theorem is expanded to cover more general models of returns and trading schemes, including the conditional expectation case where mean returns are linear in some observable features, subspace constraints (i.e., dimensionality reduction), and hedging constraints.

Portfolio Management
Chernoff-Hoeffding Inequality and Applications

Jeff M. Phillips

DOI: https://doi.org/10.48550/arXiv.1209.6396

2013-02-19

Abstract:When dealing with modern big data sets, a very common theme is reducing the set through a random process. These generally work by making "many simple estimates" of the full data set, and then judging them as a whole. Perhaps magically, these "many simple estimates" can provide a very accurate and small representation of the large data set. The key tool in showing how many of these simple estimates are needed for a fixed accuracy trade-off is the Chernoff-Hoeffding inequality[Che52,Hoe63]. This document provides a simple form of this bound, and two examples of its use.

Data Structures and Algorithms,Databases
Information Theoretic Lower Bounds for Information Theoretic Upper Bounds

Roi Livni

2024-01-14

Abstract:We examine the relationship between the mutual information between the output model and the empirical sample and the generalization of the algorithm in the context of stochastic convex optimization. Despite increasing interest in information-theoretic generalization bounds, it is uncertain if these bounds can provide insight into the exceptional performance of various learning algorithms. Our study of stochastic convex optimization reveals that, for true risk minimization, dimension-dependent mutual information is necessary. This indicates that existing information-theoretic generalization bounds fall short in capturing the generalization capabilities of algorithms like SGD and regularized ERM, which have dimension-independent sample complexity.

Machine Learning
Multiscale second-order Poincaré inequalities in probability

Mitia Duerinckx,Antoine Gloria

DOI: https://doi.org/10.48550/arXiv.1711.03158

2019-10-10

Abstract:Consider an ergodic stationary random field $A$ on the ambient space $\mathbb R^d$. In a companion article, we introduced the notion of multiscale (first-order) functional inequalities, which extend standard functional inequalities like Poincaré, covariance, and logarithmic Sobolev inequalities in the probability space, while still ensuring strong concentration properties. We also developed a constructive approach to these functional inequalities, proving their validity for prototypical examples including Gaussian fields, Poisson random tessellations, and random sequential adsorption (RSA) models, which do not satisfy standard functional inequalities. In the present contribution, we turn to second-order Poincaré inequalities à la Chatterjee: while first-order inequalities quantify the distance to constants for nonlinear functions $Z(A)$ in terms of their local dependence on the random field $A$, second-order inequalities quantify their distance to normality. For the above-mentioned examples, we prove the validity of suitable multiscale second-order Poincaré inequalities. In particular, applied to RSA models, these functional inequalities allow to complete and improve previous results by Schreiber, Penrose, and Yukich on the jamming limit, and to propose and fully analyze a more efficient algorithm to approximate the latter.

Probability
Information Theoretic Inequalities As Bounds in Superconformal Field Theory

Yang Zhou

DOI: https://doi.org/10.1142/s0217732322502443

2022-01-01

Modern Physics Letters A

Abstract:In this paper, an information theoretic approach to bounds in superconformal field theories is proposed. It is proved that the supersymmetric Renyi entropy (S) over bar alpha is a monotonically decreasing function of alpha and (alpha - 1) (S) over bar (alpha) is a concave function of alpha. Under the assumption that the thermal entropy associated with the "replica trick" time circle is bounded from below by the charge at alpha -> infinity, it is further proved that both alpha-1/alpha (S) over bar (alpha) and (alpha-1) S-alpha(& macr;) monotonically increase as functions of alpha. Because S & macr;alpha enjoys universal relations with the Weyl anomaly coefficients in even-dimensional superconformal field theories, one therefore obtains a set of bounds on these coefficients by imposing the inequalities of (S) over bar (alpha). Some of the bounds coincide with Hofman-Maldacena bounds and the others are new. We also check the inequalities for examples in odd-dimensions.
No eleventh conditional Ingleton inequality

Tobias Boege

DOI: https://doi.org/10.1080/10586458.2023.2294827

2024-02-21

Abstract:A rational probability distribution on four binary random variables $X, Y, Z, U$ is constructed which satisfies the conditional independence relations $[X \mathrel{\text{$\perp\mkern-10mu\perp$}} Y]$, $[X \mathrel{\text{$\perp\mkern-10mu\perp$}} Z \mid U]$, $[Y \mathrel{\text{$\perp\mkern-10mu\perp$}} U \mid Z]$ and $[Z \mathrel{\text{$\perp\mkern-10mu\perp$}} U \mid XY]$ but whose entropy vector violates the Ingleton inequality. This settles a recent question of Studený (IEEE Trans. Inf. Theory vol. 67, no. 11) and shows that there are, up to symmetry, precisely ten inclusion-minimal sets of conditional independence assumptions on four discrete random variables which make the Ingleton inequality hold. The last case in the classification of which of these inequalities are essentially conditional is also settled.

Information Theory,Probability
On Information (pseudo) Metric

Pierre Baudot

DOI: https://doi.org/10.48550/arXiv.2103.02008

2021-03-02

Probability

Abstract:This short note revisit information metric, underlining that it is a pseudo metric on manifolds of observables (random variables), rather than as usual on probability laws. Geodesics are characterized in terms of their boundaries and conditional independence condition. Pythagorean theorem is given, providing in special case potentially interesting natural integer triplets. This metric is computed for illustration on Diabetes dataset using infotopo package.
Some informational inequalities involving generalized trigonometric functions and a new class of generalized moments

David Puertas-Centeno,Steeve Zozor

2024-11-01

Abstract:In this work, we define a family of probability densities involving the generalized trigonometric functions defined by Drábek and Manásevich, we name Generalized Trigonometric Densities. We show their relationship with the generalized stretched Gaussians and other types of laws such as logistic, hyperbolic secant, and raised cosine probability densities. We prove that, for a fixed generalized Fisher information, this family of densities is of minimal Rényi entropy. Moreover, we introduce generalized moments via the mean of the power of a deformed cumulative distribution. The latter is defined as a cumulative of the power of the probability density function, this second parameter tuning the tail weight of the deformed cumulative distribution. These generalized moments coincide with the usual moments of a deformed probability distribution with a regularized tail. We show that, for any bounded probability density, there exists a critical value for this second parameter below which the whole subfamily of generalized moments is finite for any positive value of the first parameter (power of the moment). In addition, we show that such generalized moments satisfy remarkable properties like order relation w.r.t. the first parameter, or adequate scaling behavior. Finally, we highlight that, if we constrain such a generalized moment, both the Rényi entropy and generalized Fisher information achieve respectively their maximum and minimum for the generalized trigonometric densities.

Mathematical Physics

When the Cramer-Rao Inequality provides no information

Some families of Jensen-like inequalities with application to information theory

The Cramér‐Rao Lower Bound on Variance: Adam and Eve's “uncertainty Principle”

A constrained risk inequality for general losses

A Matrix Bernstein Inequality for Strong Rayleigh Distributions

A Note on the Harnack Inequality Related with the Martin Boundary

Information-Theoretic Perspectives on Brascamp-Lieb Inequality and Its Reverse

Information inequality problem over set functions

The One Dimensional Free Poincaré Inequality

Brascamp-Lieb Inequality and Its Reverse: An Information Theoretic View

Transportation-information Inequalities for Markov Processes

Matrix Infinitely Divisible Series: Tail Inequalities and Their Applications

A Matrix Version of Chernoff Inequality

Bounds on Portfolio Quality

Chernoff-Hoeffding Inequality and Applications

Information Theoretic Lower Bounds for Information Theoretic Upper Bounds

Multiscale second-order Poincaré inequalities in probability

Information Theoretic Inequalities As Bounds in Superconformal Field Theory

No eleventh conditional Ingleton inequality

On Information (pseudo) Metric

Some informational inequalities involving generalized trigonometric functions and a new class of generalized moments