Entropy-power inequality for weighted entropy

Yuri Suhov,Salimeh Yasaei Sekeh,Mark Kelbert

DOI: https://doi.org/10.48550/arXiv.1502.02188

2015-02-07

Information Theory

Abstract:We analyse an analog of the entropy-power inequality for the weighted entropy.

What problem does this paper attempt to address?

Weighted entropy: basic inequalities

Mark Kelbert,Izabella Stuhl,Yuri Suhov

DOI: https://doi.org/10.48550/arXiv.1710.10798

2017-10-30

Probability

Abstract:This paper represents an extended version of an earlier note [10]. The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. We analyse analogs of the Fisher information inequality and entropy power inequality for the weighted entropy and discuss connections with weighted Lieb's splitting inequality. The concepts of rates of the weighted entropy and information are also discussed.
Basic inequalities for weighted entropies

Yuri Suhov,Izabella Stuhl,Salimeh Yasaei Sekeh,Mark Kelbert

DOI: https://doi.org/10.1007/s00010-015-0396-5

2016-01-13

Aequationes Mathematicae

Abstract:The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. In this paper, we establish a number of simple inequalities for the weighted entropies (general as well as specific), mirroring similar bounds on standard (Shannon) entropies and related quantities. The required assumptions are written in terms of various expectations of weight functions. Examples are weighted Ky Fan and weighted Hadamard inequalities involving determinants of positive-definite matrices, and weighted Cramér-Rao inequalities involving the weighted Fisher information matrix.

mathematics, applied
Quantum weighted entropy and its properties

Y. Suhov,S. Zohren

DOI: https://doi.org/10.48550/arXiv.1411.0892

2014-11-04

Quantum Physics

Abstract:We introduce quantum weighted entropy in analogy to an earlier notion of (classical) weighted entropy and derive many of its properties. These include the subadditivity, concavity and strong subadditivity property of quantum weighted entropy, as well as an analog of the Araki-Lieb inequality. Interesting byproducts of the proofs are a weighted analog of Klein's inequality and non-negativity of quantum weighted relative entropy. A main difficulty is the fact that the weights in general do not commute with the density matrices.
Weighted $p$-Rényi Entropy Power Inequality: Information Theory to Quantum Shannon Theory

Junseo Lee,Hyeonjun Yeo,Kabgyun Jeong

DOI: https://doi.org/10.1007/s10773-023-05512-8

2023-11-11

Abstract:We study the $p$-Rényi entropy power inequality with a weight factor $t$ on two independent continuous random variables $X$ and $Y$. The extension essentially relies on a modulation on the sharp Young's inequality due to Bobkov and Marsiglietti. Our research provides a key result that can be used as a fundamental research finding in quantum Shannon theory, as it offers a Rényi version of the entropy power inequality for quantum systems.

Quantum Physics
Entropy numbers of embedding operators of weighted Sobolev spaces with weights that are functions of distance from some h-set

A.A. Vasil'eva

DOI: https://doi.org/10.48550/arXiv.1503.00144

2015-02-28

Functional Analysis

Abstract:In this paper order estimates for entropy numbers of embeddings of weighted Sobolev spaces on a John domain are obtained. In addition, we obtain order estimates for entropy numbers of summation operators on trees.
Power Law and Entropy

Sachio Hirokawa,Eisuke Ito

DOI: https://doi.org/10.48550/arXiv.1504.06543

2015-04-22

Abstract:Shannon(1951) and Yavuz(1974) estimated the entropy of real documents. This note drives an upper bound of entropy from the parameter of the power law.

Information Theory
A Conditional Entropy Power Inequality for Dependent Variables

Oliver Johnson,O. Johnson

DOI: https://doi.org/10.1109/tit.2004.831790

IF: 2.5

2004-08-01

IEEE Transactions on Information Theory

Abstract:We provide a condition under which a version of Shannon's entropy power inequality will hold for dependent variables. We first provide a Fisher information inequality extending that found in the independent case. The key ingredients are a conditional expectation representation for the score function of a sum, and the de Bruijn identity which relates entropy and Fisher information.

computer science, information systems,engineering, electrical & electronic
Thinning, photonic beamsplitting, and a general discrete entropy power inequality

Saikat Guha,Jeffrey H. Shapiro,Raul Garcia-Patron Sanchez

DOI: https://doi.org/10.1109/ISIT.2016.7541390

2016-05-25

Abstract:Many partially-successful attempts have been made to find the most natural discrete-variable version of Shannon's entropy power inequality (EPI). We develop an axiomatic framework from which we deduce the natural form of a discrete-variable EPI and an associated entropic monotonicity in a discrete-variable central limit theorem. In this discrete EPI, the geometric distribution, which has the maximum entropy among all discrete distributions with a given mean, assumes a role analogous to the Gaussian distribution in Shannon's EPI. The entropy power of $X$ is defined as the mean of a geometric random variable with entropy $H(X)$. The crux of our construction is a discrete-variable version of Lieb's scaled addition $X \boxplus_\eta Y$ of two discrete random variables $X$ and $Y$ with $\eta \in (0, 1)$. We discuss the relationship of our discrete EPI with recent work of Yu and Johnson who developed an EPI for a restricted class of random variables that have ultra-log-concave (ULC) distributions. Even though we leave open the proof of the aforesaid natural form of the discrete EPI, we show that this discrete EPI holds true for variables with arbitrary discrete distributions when the entropy power is redefined as $e^{H(X)}$ in analogy with the continuous version. Finally, we show that our conjectured discrete EPI is a special case of the yet-unproven Entropy Photon-number Inequality (EPnI), which assumes a role analogous to Shannon's EPI in capacity proofs for Gaussian bosonic (quantum) channels.

Information Theory,Quantum Physics
Weighted Gaussian entropy and determinant inequalities

I. Stuhl,M. Kelbert,Y. Suhov,S. Yasaei Sekeh

DOI: https://doi.org/10.1007/s00010-021-00861-3

2022-01-22

Aequationes Mathematicae

Abstract:We produce a series of results extending information-theoretical inequalities (discussed by Dembo–Cover–Thomas in 1988–1991) to a weighted version of entropy. Most of the resulting inequalities involve the Gaussian weighted entropy; they imply a number of new relations for determinants of positive-definite matrices. Unlike the Shannon entropy where the contribution of an outcome depends only upon its probability, the weighted (or context-dependent) entropy takes into account a ‘value’ of an outcome determined by a given weight function φdocumentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$${varphi }$$end{document}. An example of a new result is a weighted version of the strong Hadamard inequality (SHI) between the determinants of a positive-definite d×ddocumentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$$d imes d$$end{document} matrix and its square blocks (sub-matrices) of different sizes. When φ≡1documentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$${varphi }equiv 1$$end{document}, the weighted inequality becomes a ‘standard’ SHI; in general, the weighted version requires some assumptions upon φdocumentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$${varphi }$$end{document}. The SHI and its weighted version generalize a widely known ‘usual’ Hadamard inequality detC≤∏j=1dCjjdocumentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$${mathrm{det}},{{mathbf {C}}}le prod olimits _{j=1}^dC_{jj}$$end{document}.

mathematics, applied
Information Theoretic Proofs of Entropy Power Inequalities

Olivier Rioul

DOI: https://doi.org/10.1109/TIT.2010.2090193

2010-08-24

Abstract:While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai and Verdú used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.

Information Theory
Unifying the Brascamp-Lieb Inequality and the Entropy Power Inequality

Venkat Anantharam,Varun Jog,Chandra Nair

DOI: https://doi.org/10.48550/arXiv.1901.06619

2021-09-27

Abstract:The entropy power inequality (EPI) and the Brascamp-Lieb inequality (BLI) are fundamental inequalities concerning the differential entropies of linear transformations of random vectors. The EPI provides lower bounds for the differential entropy of linear transformations of random vectors with independent components. The BLI, on the other hand, provides upper bounds on the differential entropy of a random vector in terms of the differential entropies of some of its linear transformations. In this paper, we define a family of entropy functionals, which we show are subadditive. We then establish that Gaussians are extremal for these functionals by mimicking the idea in Geng and Nair (2014). As a consequence, we obtain a new entropy inequality that generalizes both the BLI and EPI. By considering a variety of independence relations among the components of the random vectors appearing in these functionals, we also obtain families of inequalities that lie between the EPI and the BLI.

Information Theory,Probability
Balanced distribution-energy inequalities and related entropy bounds

Michel Rumin

DOI: https://doi.org/10.1215/00127094-1444305

2010-08-10

Abstract:Let $A$ be a self-adjoint operator acting over a space $X$ endowed with a partition. We give lower bounds on the energy of a mixed state $\rho$ from its distribution in the partition and the spectral density of $A$. These bounds improve with the refinement of the partition, and generalize inequalities by Li-Yau and Lieb--Thirring for the Laplacian in $\R^n$. They imply an uncertainty principle, giving a lower bound on the sum of the spatial entropy of $\rho$, as seen from $X$, and some spectral entropy, with respect to its energy distribution. On $\R^n$, this yields lower bounds on the sum of the entropy of the densities of $\rho$ and its Fourier transform. A general log-Sobolev inequality is also shown. It holds on mixed states, without Markovian or positivity assumption on $A$.

Functional Analysis,Information Theory,Spectral Theory
Error Bounds on a Mixed Entropy Inequality

James Melbourne,Saurav Talukdar,Shreyas Bhaban,Murti V. Salapaka

DOI: https://doi.org/10.1109/isit.2018.8437601

2018-06-01

Abstract:Motivated by the entropy computations relevant to the evaluation of decrease in entropy in bit reset operations, the authors investigate the deficit in an entropic inequality involving two independent random variables, one continuous and the other discrete. In the case where the continuous random variable is Gaussian, we derive strong quantitative bounds on the deficit in the inequality. More explicitly it is shown that the decay of the deficit is sub-Gaussian with respect to the reciprocal of the standard deviation of the Gaussian variable. What is more, up to rational terms these results are shown to be sharp.
A generalization of Costa's Entropy Power Inequality

Luca Tamanini

DOI: https://doi.org/10.48550/arXiv.2012.12230

2020-12-22

Information Theory

Abstract:Aim of this short note is to study Shannon's entropy power along entropic interpolations, thus generalizing Costa's concavity theorem. We shall provide two proofs of independent interest: the former by $\Gamma$-calculus, hence applicable to more abstract frameworks; the latter with an explicit remainder term, reminiscent of [20], allowing us to characterize the case of equality.
Quantum Rényi Entropy Functionals for Bosonic Gaussian Systems

Junseo Lee,Kabgyun Jeong

DOI: https://doi.org/10.1016/j.physleta.2023.129183

2023-10-24

Abstract:In this study, the quantum Rényi entropy power inequality of order $p>1$ and power $\kappa$ is introduced as a quantum analog of the classical Rényi-$p$ entropy power inequality. To derive this inequality, we first exploit the Wehrl-$p$ entropy power inequality on bosonic Gaussian systems via the mixing operation of quantum convolution, which is a generalized beam-splitter operation. This observation directly provides a quantum Rényi-$p$ entropy power inequality over a quasi-probability distribution for $D$-mode bosonic Gaussian regimes. The proposed inequality is expected to be useful for the nontrivial computing of quantum channel capacities, particularly universal upper bounds on bosonic Gaussian quantum channels, and a Gaussian entanglement witness in the case of Gaussian amplifier via squeezing operations.

Quantum Physics
Weighted information and weighted entropic inequalities for qutrit and ququart states

V. I. Man'ko,Z. Seilov

DOI: https://doi.org/10.48550/arXiv.1604.01089

2016-04-04

Quantum Physics

Abstract:The notion of weighted quantum entropy is reviewed and considered for bipartite and noncomposite quantum systems. The known for the weighted entropy information inequality (subadditivity condition) is extended to the case of indivisible qudit system on example of qutrit. This new inequality for qutrit density matrix is discussed for different cases of weights and states. The role of weighted entropy is discussed in connection with studies of nonlinear quantum channels.
Entropy Inequalities and Gaussian Comparisons

Efe Aras,Thomas A. Courtade

DOI: https://doi.org/10.48550/arXiv.2206.14182

2022-06-28

Information Theory

Abstract:We establish a general class of entropy inequalities that take the concise form of Gaussian comparisons. The main result unifies many classical and recent results, including the Shannon-Stam inequality, the Brunn-Minkowski inequality, the Zamir-Feder inequality, the Brascamp-Lieb and Barthe inequalities, the Anantharam-Jog-Nair inequality, and others.
Results on the solutions of maximum weighted Renyi entropy problems

Salimeh Yasaei Sekeh

DOI: https://doi.org/10.48550/arXiv.1510.07461

2015-10-26

Information Theory

Abstract:In this paper, following standard arguments, the maximum Renyi entropy problem for the weighted case is analyzed. We verify that under some constrains on weight function, the Student-r and Student-t distributions maximize the weighted Renyi entropy. Furthermore, an extended version of the Hadamard inequality is derived.
Weighted Approach to Rényi Entropy

Marek Śmieja,Jacek Tabor

DOI: https://doi.org/10.48550/arXiv.1204.0075

2012-03-31

Information Theory

Abstract:R\'enyi entropy of order \alpha is a general measure of entropy. In this paper we derive estimations for the R\'enyi entropy of the mixture of sources in terms of the entropy of the single sources. These relations allow to compute the R\'enyi entropy dimension of arbitrary order of a mixture of measures. The key for obtaining these results is our new definition of the weighted R\'enyi entropy. It is shown that weighted entropy is equal to the classical R\'enyi entropy.
Entropy numbers of weighted Sobolev classes on the unit sphere

Heping Wang,Wang Kai

DOI: https://doi.org/10.48550/arXiv.1912.03451

2019-12-07

Classical Analysis and ODEs

Abstract:We obtain the asymptotic orders of entropy numbers of Sobolev classes on the unit sphere with Dunkl weight which associates with the finite reflection group. Moreover, the asymptotic order of entropy numbers of weighted Sobolev classes on the unit ball and on the standard simplex are discussed.

Entropy-power inequality for weighted entropy

Weighted entropy: basic inequalities

Basic inequalities for weighted entropies

Quantum weighted entropy and its properties

Weighted $p$-Rényi Entropy Power Inequality: Information Theory to Quantum Shannon Theory

Entropy numbers of embedding operators of weighted Sobolev spaces with weights that are functions of distance from some h-set

Power Law and Entropy

A Conditional Entropy Power Inequality for Dependent Variables

Thinning, photonic beamsplitting, and a general discrete entropy power inequality

Weighted Gaussian entropy and determinant inequalities

Information Theoretic Proofs of Entropy Power Inequalities

Unifying the Brascamp-Lieb Inequality and the Entropy Power Inequality

Balanced distribution-energy inequalities and related entropy bounds

Error Bounds on a Mixed Entropy Inequality

A generalization of Costa's Entropy Power Inequality

Quantum Rényi Entropy Functionals for Bosonic Gaussian Systems

Weighted information and weighted entropic inequalities for qutrit and ququart states

Entropy Inequalities and Gaussian Comparisons

Results on the solutions of maximum weighted Renyi entropy problems

Weighted Approach to Rényi Entropy

Entropy numbers of weighted Sobolev classes on the unit sphere