A simple proof of the entropy-power inequality

S. Verdu,D. Guo

DOI: https://doi.org/10.1109/TIT.2006.872978

IF: 2.5

2006-01-01

IEEE Transactions on Information Theory

Abstract:This correspondence gives a simple proof of Shannon's entropy-power inequality (EPI) using the relationship between mutual information and minimum mean-square error (MMSE) in Gaussian channels.

What problem does this paper attempt to address?

Proof of Entropy Power Inequalities Via MMSE.

Dongning Guo,Shlomo (Shitz) Shamai,Sergio Verdu

DOI: https://doi.org/10.1109/isit.2006.261880

2006-01-01

Abstract:The differential entropy of a random variable (or vector) can be expressed as the integral over signal-to-noise ratio (SNR) of the minimum mean-square error (MMSE) of estimating the variable (or vector) when observed in additive Gaussian noise. This representation sidesteps Fisher's information to provide simple and insightful proofs for Shannon's entropy power inequality (EPI) and two of its variations: Costa's strengthened EPI in the case in which one of the variables is Gaussian, and a generalized EPI for linear transformations of a random vector due to Zamir and Feder
Information Theoretic Proofs of Entropy Power Inequalities

Olivier Rioul

DOI: https://doi.org/10.1109/TIT.2010.2090193

2010-08-24

Abstract:While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai and Verdú used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.

Information Theory
Continuity of mutual entropy in the large signal-to-noise ratio limit

Mark Kelbert,Yuri Suhov

DOI: https://doi.org/10.48550/arXiv.0911.1275

2013-04-03

Abstract:This article addresses the issue of the proof of the entropy power inequality (EPI), an important tool in the analysis of Gaussian channels of information transmission, proposed by Shannon. We analyse continuity properties of the mutual entropy of the input and output signals in an additive memoryless channel and discuss assumptions under which the entropy-power inequality holds true.

Information Theory,Information Retrieval,Probability
A New Extremal Entropy Inequality With Applications

Hon Fah Chong,Ying-Chang Liang

DOI: https://doi.org/10.1109/ISIT.2013.6620298

2013-01-01

Abstract:Liu et al. proved an extremal entropy inequality using a vector generalization of the Costa entropy-power inequality (EPI). The generalized Costa EPI was proved, in turn, using a perturbation approach via a fundamental relationship between the derivative of mutual information and the minimum mean-square error (MMSE) estimate in linear vector Gaussian channels. In this paper, we consider two new variations of the (Liu et al.) extremal entropy inequality. Instead of employing perturbation approaches, we employ a new method recently introduced by Geng and Nair, which was used to resolve the capacity region of the Gaussian MIMO broadcast channel (BC) with common and private messages. As an application, we use one of the extremal entropy inequalities to prove the capacity region of a class of reversely degraded Gaussian MIMO BC with three users and three-degraded message sets.
Thinning, photonic beamsplitting, and a general discrete entropy power inequality

Saikat Guha,Jeffrey H. Shapiro,Raul Garcia-Patron Sanchez

DOI: https://doi.org/10.1109/ISIT.2016.7541390

2016-05-25

Abstract:Many partially-successful attempts have been made to find the most natural discrete-variable version of Shannon's entropy power inequality (EPI). We develop an axiomatic framework from which we deduce the natural form of a discrete-variable EPI and an associated entropic monotonicity in a discrete-variable central limit theorem. In this discrete EPI, the geometric distribution, which has the maximum entropy among all discrete distributions with a given mean, assumes a role analogous to the Gaussian distribution in Shannon's EPI. The entropy power of $X$ is defined as the mean of a geometric random variable with entropy $H(X)$. The crux of our construction is a discrete-variable version of Lieb's scaled addition $X \boxplus_\eta Y$ of two discrete random variables $X$ and $Y$ with $\eta \in (0, 1)$. We discuss the relationship of our discrete EPI with recent work of Yu and Johnson who developed an EPI for a restricted class of random variables that have ultra-log-concave (ULC) distributions. Even though we leave open the proof of the aforesaid natural form of the discrete EPI, we show that this discrete EPI holds true for variables with arbitrary discrete distributions when the entropy power is redefined as $e^{H(X)}$ in analogy with the continuous version. Finally, we show that our conjectured discrete EPI is a special case of the yet-unproven Entropy Photon-number Inequality (EPnI), which assumes a role analogous to Shannon's EPI in capacity proofs for Gaussian bosonic (quantum) channels.

Information Theory,Quantum Physics
Mutual Information and Minimum Mean-square Error in Gaussian Channels

Dn Guo,S Shamai,S Verdu

DOI: https://doi.org/10.1109/tit.2005.844072

IF: 2.5

2005-01-01

IEEE Transactions on Information Theory

Abstract:This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation.This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose SNR is chosen uniformly distributed between 0 and SNR.
A Conditional Entropy Power Inequality for Dependent Variables

Oliver Johnson,O. Johnson

DOI: https://doi.org/10.1109/tit.2004.831790

IF: 2.5

2004-08-01

IEEE Transactions on Information Theory

Abstract:We provide a condition under which a version of Shannon's entropy power inequality will hold for dependent variables. We first provide a Fisher information inequality extending that found in the independent case. The key ingredients are a conditional expectation representation for the score function of a sum, and the de Bruijn identity which relates entropy and Fisher information.

computer science, information systems,engineering, electrical & electronic
Entropy-power inequality for weighted entropy

Yuri Suhov,Salimeh Yasaei Sekeh,Mark Kelbert

DOI: https://doi.org/10.48550/arXiv.1502.02188

2015-02-07

Information Theory

Abstract:We analyse an analog of the entropy-power inequality for the weighted entropy.
Unifying the Brascamp-Lieb Inequality and the Entropy Power Inequality

Venkat Anantharam,Varun Jog,Chandra Nair

DOI: https://doi.org/10.48550/arXiv.1901.06619

2021-09-27

Abstract:The entropy power inequality (EPI) and the Brascamp-Lieb inequality (BLI) are fundamental inequalities concerning the differential entropies of linear transformations of random vectors. The EPI provides lower bounds for the differential entropy of linear transformations of random vectors with independent components. The BLI, on the other hand, provides upper bounds on the differential entropy of a random vector in terms of the differential entropies of some of its linear transformations. In this paper, we define a family of entropy functionals, which we show are subadditive. We then establish that Gaussians are extremal for these functionals by mimicking the idea in Geng and Nair (2014). As a consequence, we obtain a new entropy inequality that generalizes both the BLI and EPI. By considering a variety of independence relations among the components of the random vectors appearing in these functionals, we also obtain families of inequalities that lie between the EPI and the BLI.

Information Theory,Probability
The entropy power conjecture implies the McKean conjecture

Guillaume Wang

2024-08-26

Abstract:After reviewing the entropy power, the McKean, and the Gaussian completely monotone conjectures, we prove that the first implies the second, for each order of the time-derivative. The proof is elementary and is based on manipulating the Bell polynomials.

Information Theory
A Marginal Characterization of Entropy Functions for Conditional Mutually Independent Random Variables (with Application to Wyner's Common Information)

Qi Chen,Fan Cheng,Tie Liu,Raymond W. Yeung

DOI: https://doi.org/10.1109/isit.2015.7282600

2015-01-01

Abstract:We prove that by imposing a conditional mutual independence constraint and a marginalisation constraint, the almost entropic region can be completely characterised by Shannon-type information inequalities. Such a property is applied to obtain an explicit lower bound on the generalised Wyner common information.
A generalization of Costa's Entropy Power Inequality

Luca Tamanini

DOI: https://doi.org/10.48550/arXiv.2012.12230

2020-12-22

Information Theory

Abstract:Aim of this short note is to study Shannon's entropy power along entropic interpolations, thus generalizing Costa's concavity theorem. We shall provide two proofs of independent interest: the former by $\Gamma$-calculus, hence applicable to more abstract frameworks; the latter with an explicit remainder term, reminiscent of [20], allowing us to characterize the case of equality.
The multimode conditional quantum Entropy Power Inequality and the squashed entanglement of the extreme multimode bosonic Gaussian channels

Alessandro Falco,Giacomo De Palma

2024-10-18

Abstract:We prove the multimode conditional quantum Entropy Power Inequality for bosonic quantum systems. This inequality determines the minimum conditional von Neumann entropy of the output of the most general linear mixing of bosonic quantum modes among all the input states of the modes with given conditional entropies. Bosonic quantum systems constitute the mathematical model for the electromagnetic radiation in the quantum regime, which provides the most promising platform for quantum communication and quantum key distribution. We apply our multimode conditional quantum Entropy Power Inequality to determine new lower bounds to the squashed entanglement of a large family of bosonic quantum Gaussian states. The squashed entanglement is one of the main entanglement measures in quantum communication theory, providing the best known upper bound to the distillable key. Exploiting this result, we determine a new lower bound to the squashed entanglement of the multimode bosonic Gaussian channels that are extreme, i.e., that cannot be decomposed as a non-trivial convex combination of quantum channels. The squashed entanglement of a quantum channel provides an upper bound to its secret-key capacity, i.e., the capacity to generate a secret key shared between the sender and the receiver. Lower bounds to the squashed entanglement are notoriously hard to prove. Our results contribute to break this barrier and will stimulate further research in the field of quantum communication with bosonic quantum systems.

Quantum Physics,Mathematical Physics
Entropy Inequalities and Gaussian Comparisons

Efe Aras,Thomas A. Courtade

DOI: https://doi.org/10.48550/arXiv.2206.14182

2022-06-28

Information Theory

Abstract:We establish a general class of entropy inequalities that take the concise form of Gaussian comparisons. The main result unifies many classical and recent results, including the Shannon-Stam inequality, the Brunn-Minkowski inequality, the Zamir-Feder inequality, the Brascamp-Lieb and Barthe inequalities, the Anantharam-Jog-Nair inequality, and others.
Quantum Rényi Entropy Functionals for Bosonic Gaussian Systems

Junseo Lee,Kabgyun Jeong

DOI: https://doi.org/10.1016/j.physleta.2023.129183

2023-10-24

Abstract:In this study, the quantum Rényi entropy power inequality of order $p>1$ and power $\kappa$ is introduced as a quantum analog of the classical Rényi-$p$ entropy power inequality. To derive this inequality, we first exploit the Wehrl-$p$ entropy power inequality on bosonic Gaussian systems via the mixing operation of quantum convolution, which is a generalized beam-splitter operation. This observation directly provides a quantum Rényi-$p$ entropy power inequality over a quasi-probability distribution for $D$-mode bosonic Gaussian regimes. The proposed inequality is expected to be useful for the nontrivial computing of quantum channel capacities, particularly universal upper bounds on bosonic Gaussian quantum channels, and a Gaussian entanglement witness in the case of Gaussian amplifier via squeezing operations.

Quantum Physics
Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Dongning Guo,Yihong Wu,Shlomo Shamai (Shitz),Sergio Verdu

DOI: https://doi.org/10.1109/tit.2011.2111010

IF: 2.5

2011-01-01

IEEE Transactions on Information Theory

Abstract:Consider the minimum mean-square error (MMSE) of estimating an arbitrary random variable from its observation contaminated by Gaussian noise. The MMSE can be regarded as a function of the signal-to-noise ratio (SNR) as well as a functional of the input distribution (of the random variable to be estimated). It is shown that the MMSE is concave in the input distribution at any given SNR. For a given input distribution, the MMSE is found to be infinitely differentiable at all positive SNR, and in fact a real analytic function in SNR under mild conditions. The key to these regularity results is that the posterior distribution conditioned on the observation through Gaussian channels always decays at least as quickly as some Gaussian density. Furthermore, simple expressions for the first three derivatives of the MMSE with respect to the SNR are obtained. It is also shown that, as functions of the SNR, the curves for the MMSE of a Gaussian input and that of a non-Gaussian input cross at most once over all SNRs. These properties lead to simple proofs of the facts that Gaussian inputs achieve both the secrecy capacity of scalar Gaussian wiretap channels and the capacity of scalar Gaussian broadcast channels, as well as a simple proof of the entropy power inequality in the special case where one of the variables is Gaussian.
Another short and elementary proof of strong subadditivity of quantum entropy

Mary Beth Ruskai

DOI: https://doi.org/10.1016/s0034-4877(07)00019-5

IF: 0.808

2007-08-01

Reports on Mathematical Physics

Abstract:A short and elementary proof of the joint convexity of relative entropy is presented, using nothing beyond linear algebra. The key ingredients are an easily verified integral representation and the strategy used to prove the Cauchy-Schwarz inequality in elementary courses. Several consequences are proved in a way which allows an elementary proof of strong subadditivity in a few more lines. Some expository material on Schwarz inequalities for operators and the Holevo bound for partial measurements is also included.

physics, mathematical
Basic inequalities for weighted entropies

Yuri Suhov,Izabella Stuhl,Salimeh Yasaei Sekeh,Mark Kelbert

DOI: https://doi.org/10.1007/s00010-015-0396-5

2016-01-13

Aequationes Mathematicae

Abstract:The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. In this paper, we establish a number of simple inequalities for the weighted entropies (general as well as specific), mirroring similar bounds on standard (Shannon) entropies and related quantities. The required assumptions are written in terms of various expectations of weight functions. Examples are weighted Ky Fan and weighted Hadamard inequalities involving determinants of positive-definite matrices, and weighted Cramér-Rao inequalities involving the weighted Fisher information matrix.

mathematics, applied
An Information-Theoretic Proof of a Hypercontractive Inequality

Ehud Friedgut

DOI: https://doi.org/10.3390/e26110966

IF: 2.738

2024-11-13

Entropy

Abstract:The famous hypercontractive estimate discovered independently by Gross, Bonami and Beckner has had a great impact on combinatorics and theoretical computer science since it was first used in this setting in a seminal paper by Kahn, Kalai and Linial. The usual proofs of this inequality begin with two-point space, where some elementary calculus is used and then generalised immediately by introducing another dimension using submultiplicativity (Minkowski's integral inequality). In this paper, we prove this inequality using information theory. We compare the entropy of a pair of correlated vectors in {0,1}n to their separate entropies, analysing them bit by bit (not as a figure of speech, but as the bits are revealed) using the chain rule of entropy.

physics, multidisciplinary
Weighted $p$-Rényi Entropy Power Inequality: Information Theory to Quantum Shannon Theory

Junseo Lee,Hyeonjun Yeo,Kabgyun Jeong

DOI: https://doi.org/10.1007/s10773-023-05512-8

2023-11-11

Abstract:We study the $p$-Rényi entropy power inequality with a weight factor $t$ on two independent continuous random variables $X$ and $Y$. The extension essentially relies on a modulation on the sharp Young's inequality due to Bobkov and Marsiglietti. Our research provides a key result that can be used as a fundamental research finding in quantum Shannon theory, as it offers a Rényi version of the entropy power inequality for quantum systems.

Quantum Physics

A simple proof of the entropy-power inequality

Proof of Entropy Power Inequalities Via MMSE.

Information Theoretic Proofs of Entropy Power Inequalities

Continuity of mutual entropy in the large signal-to-noise ratio limit

A New Extremal Entropy Inequality With Applications

Thinning, photonic beamsplitting, and a general discrete entropy power inequality

Mutual Information and Minimum Mean-square Error in Gaussian Channels

A Conditional Entropy Power Inequality for Dependent Variables

Entropy-power inequality for weighted entropy

Unifying the Brascamp-Lieb Inequality and the Entropy Power Inequality

The entropy power conjecture implies the McKean conjecture

A Marginal Characterization of Entropy Functions for Conditional Mutually Independent Random Variables (with Application to Wyner's Common Information)

A generalization of Costa's Entropy Power Inequality

The multimode conditional quantum Entropy Power Inequality and the squashed entanglement of the extreme multimode bosonic Gaussian channels

Entropy Inequalities and Gaussian Comparisons

Quantum Rényi Entropy Functionals for Bosonic Gaussian Systems

Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Another short and elementary proof of strong subadditivity of quantum entropy

Basic inequalities for weighted entropies

An Information-Theoretic Proof of a Hypercontractive Inequality

Weighted $p$-Rényi Entropy Power Inequality: Information Theory to Quantum Shannon Theory