A note on Shannon entropy

Tomasz Sobieszek

DOI: https://doi.org/10.48550/arXiv.1203.3322

2012-03-15

Abstract:We present a somewhat different way of looking on Shannon entropy. This leads to an axiomatisation of Shannon entropy that is essentially equivalent to that of Fadeev. In particular we give a new proof of Fadeev theorem.

Information Theory

What problem does this paper attempt to address?

Shannon's entropy revisited

Sergei Viznyuk

DOI: https://doi.org/10.48550/arXiv.1504.01407

2015-03-19

Abstract:I consider the effect of a finite sample size on the entropy of a sample of independent events. I propose formula for entropy which satisfies Shannon's axioms, and which reduces to Shannon's entropy when sample size is infinite. I discuss the physical meaning of the difference between two formulas, including some practical implications, such as maximum achievable channel utilization, and minimum achievable communication protocol overhead, for a given message size.

Information Theory
Generalized Shannon-Khinchin Axioms and Uniqueness Theorem for Pseudo-additive Entropies

Velimir M. Ilic,Miomir S. Stankovic

DOI: https://doi.org/10.1016/j.physa.2014.05.009

2013-11-02

Abstract:We consider the Shannon-Khinchin axiomatic systems for the characterization of generalized entropies such as Sharma-Mital and Frank-Daffertshofer entropy. We provide the generalization of Shannon-Khinchin axioms and give the corresponding uniqueness theorem. The previous attempts at such axiomatizations are also discussed.

Mathematical Physics
A generalization of Costa's Entropy Power Inequality

Luca Tamanini

DOI: https://doi.org/10.48550/arXiv.2012.12230

2020-12-22

Information Theory

Abstract:Aim of this short note is to study Shannon's entropy power along entropic interpolations, thus generalizing Costa's concavity theorem. We shall provide two proofs of independent interest: the former by $\Gamma$-calculus, hence applicable to more abstract frameworks; the latter with an explicit remainder term, reminiscent of [20], allowing us to characterize the case of equality.
Axioms and uniqueness theorem for Tsallis entropy

Sumiyoshi Abe

DOI: https://doi.org/10.1016/S0375-9601%2800%2900337-6

2000-06-01

Abstract:The Shannon-Khinchin axioms for the ordinary information entropy are generalized in a natural way to the nonextensive systems based on the concept of nonextensive conditional entropy, and a complete proof of the uniqueness theorem for the Tsallis entropy is presented.

Statistical Mechanics
Generalization of Shannon-Khinchin Axioms to Nonextensive Systems and the Uniqueness Theorem

Hiroki Suyari

DOI: https://doi.org/10.48550/arXiv.math-ph/0205004

2002-05-29

Abstract:The Shannon-Khinchin axioms are generalized to nonextensive systems and the uniqueness theorem for the nonextensive entropy is proved rigorously. In the present axioms, Shannon additivity is used as additivity in contrast to pseudoadditivity in Abe's axioms. The results reveal that Tsallis entropy is the simplest among all nonextensive entropies which can be obtained from the generalized Shannon-Khinchin axioms.

Mathematical Physics
Tsallis entropy and generalized Shannon additivity

Sonja Jäckle,Karsten Keller

DOI: https://doi.org/10.48550/arXiv.1704.08011

2017-04-26

Abstract:The Tsallis entropy given for a positive parameter $\alpha$ can be considered as a modification of the classical Shannon entropy. For the latter, corresponding to $\alpha=1$, there exist many axiomatic characterizations. One of them based on the well-known Khinchin-Shannon axioms has been simplified several times and adapted to Tsallis entropy, where the axiom of (generalized) Shannon additivity is playing a central role. The main aim of this paper is to discuss this axiom in the context of Tsallis entropy. We show that it is sufficient for characterizing Tsallis entropy with the exceptions of cases $\alpha=1,2$ discussed separately.

Mathematical Physics
Shannon's entropy and its bounds for some a priori known equiprobable states

Symeonidis, Eleutherius,Mitroi-Symeonidis, Flavia-Corina

DOI: https://doi.org/10.1007/s00010-024-01068-y

2024-05-16

Aequationes Mathematicae

Abstract:It is known that Shannon's entropy is nonnegative and its maximum value is reached for equiprobable events. Adding or removing impossible events does not affect Shannon's entropy. However, if we increase the number of events and consider not necessarily all of them equiprobable, but at least as many of them as the initial number of equiprobable events, how does Shannon's entropy change? We study the lower bound of the interval where the probability value of the a priori assumed equiprobable states must belong when the entropy increases.

mathematics, applied
Equivalence of additive-combinatorial linear inequalities for Shannon entropy and differential entropy

Ashok Vardhan Makkuva,Yihong Wu

DOI: https://doi.org/10.48550/arXiv.1601.07498

2016-09-09

Abstract:This paper addresses the correspondence between linear inequalities of Shannon entropy and differential entropy for sums of independent group-valued random variables. We show that any balanced (with the sum of coefficients being zero) linear inequality of Shannon entropy holds if and only if its differential entropy counterpart also holds; moreover, any linear inequality for differential entropy must be balanced. In particular, our result shows that recently proved differential entropy inequalities by Kontoyiannis and Madiman \cite{KM14} can be deduced from their discrete counterparts due to Tao \cite{Tao10} in a unified manner. Generalizations to certain abelian groups are also obtained. Our proof of extending inequalities of Shannon entropy to differential entropy relies on a result of Rényi \cite{Renyi59} which relates the Shannon entropy of a finely discretized random variable to its differential entropy and also helps in establishing the entropy of the sum of quantized random variables is asymptotically equal to that of the quantized sum; the converse uses the asymptotics of the differential entropy of convolutions with weak additive noise.

Information Theory
Comment on "Stronger subadditivity of entropy" by Lieb and Seiringer <a class="link-https" data-arxiv-id="math-ph/0412009" href="https://arxiv.org/abs/math-ph/0412009">math-ph/0412009</a>

Mary Beth Ruskai

DOI: https://doi.org/10.48550/arXiv.math-ph/0603022

2006-04-05

Abstract:We show how recent results of Lieb and Seiringer [<a class="link-https" data-arxiv-id="math-ph/0412009" href="https://arxiv.org/abs/math-ph/0412009">math-ph/0412009</a>; Phys. Rev. A 71, 062329 (2005)] can be obtained from repeated use of the monotonicity of relative entropy under partial traces, and explain how to use their approach to obtain tighter bounds in many situations.

Mathematical Physics,Quantum Physics
A characterization of von Neumann entropy using functors

K. Nakahira

2023-09-19

Abstract:Baez, Fritz, and Leinster derived a method for characterizing Shannon entropy in classical systems. In this method, they considered a functor from a certain category to the monoid of non-negative real numbers with addition as a map from measure-preserving functions to non-negative real numbers, and derived Shannon entropy by imposing several simple conditions. We propose a method for characterizing von Neumann entropy by extending their results to quantum systems.

Quantum Physics
Nonsymmetric entropy I: basic concepts and results

Chengshi Liu

DOI: https://doi.org/10.48550/arXiv.cs/0611038

2006-11-09

Information Theory

Abstract:A new concept named nonsymmetric entropy which generalizes the concepts of Boltzman's entropy and shannon's entropy, was introduced. Maximal nonsymmetric entropy principle was proven. Some important distribution laws were derived naturally from maximal nonsymmetric entropy principle.
On a recent conjecture by Z. Van Herstraeten and N.J. Cerf for the quantum Wigner entropy

Nuno Costa Dias,João Nuno Prata

2023-03-19

Abstract:We address a recent conjecture stated by Z. Van Herstraeten and N.J. Cerf. They claim that the Shannon entropy for positive Wigner functions is bounded below by a positive constant, which can be attained only by Gaussian pure states. We introduce an alternative definition of entropy for all absolutely integrable Wigner functions, which is the Shannon entropy for positive Wigner functions. Moreover, we are able to prove, in arbitrary dimension, that this entropy is indeed bounded below by a positive constant, which is not very distant from the constant suggested by Van Herstraeten and Cerf. We also prove an analogous result for another conjecture stated by the same authors for the Rényi entropy of positive Wigner functions. As a by-product we prove a new inequality for the radar-ambiguity function (and for the Wigner distribution) which is reminiscent of Lieb's inequalities.

Quantum Physics,Mathematical Physics
A unified formulation of entropy and its application

Narayanaswamy Balakrishnan,Francesco Buono,Maria Longobardi

DOI: https://doi.org/10.1016/j.physa.2022.127214

2022-06-01

Abstract:In this paper, a general formulation of entropy is proposed. It depends on two parameters and includes Shannon, Tsallis and fractional entropy, all as special cases. This measure of information is referred to as fractional Tsallis entropy and some of its properties are then studied. Furthermore, the corresponding entropy in the context of Dempster-Shafer theory of evidence is proposed and referred to as fractional version of Tsallis-Deng entropy. Finally, an application to two classification problems is presented.
A Characterization of Entropy as a Universal Monoidal Natural Transformation

Cheuk Ting Li

2024-04-14

Abstract:We show that the essential properties of entropy (monotonicity, additivity and subadditivity) are consequences of entropy being a monoidal natural transformation from the under category functor $-/\mathsf{LProb}_{\rho}$ (where $\mathsf{LProb}_{\rho}$ is category of $\rho$-th-power-summable probability distributions, $0<\rho<1$) to $\Delta_{\mathbb{R}}$. Moreover, the Shannon entropy can be characterized as the universal monoidal natural transformation from $-/\mathsf{LProb}_{\rho}$ to the category of integrally closed partially ordered abelian groups (a reflective subcategory of the lax-slice 2-category over $\mathsf{MonCat}_{\ell}$ in the 2-category of monoidal categories), providing a succinct characterization of Shannon entropy as a reflection arrow. We can likewise define entropy for every monoidal category with a monoidal structure on its under categories (e.g. the category of finite abelian groups, the category of finite inhabited sets, the category of finite dimensional vector spaces, and the augmented simplex category) via the reflection arrow. This implies that all these entropies over different categories are components of a single natural transformation (the unit of the idempotent monad), allowing us to connect these entropies in a natural manner. We also provide a universal characterization of the conditional Shannon entropy based on the chain rule which, unlike the characterization of information loss by Baez, Fritz and Leinster, does not require any continuity assumption.

Category Theory,Information Theory
Entropy, virtual Abelianness and Shannon orbit equivalence

DAVID KERR,HANFENG LI

DOI: https://doi.org/10.1017/etds.2024.26

2024-04-04

Ergodic Theory and Dynamical Systems

Abstract:We prove that if two free probability-measure-preserving (p.m.p.) -actions are Shannon orbit equivalent, then they have the same entropy. The argument also applies more generally to yield the same conclusion for free p.m.p. actions of finitely generated virtually Abelian groups. Together with the isomorphism theorems of Ornstein and Ornstein–Weiss and the entropy invariance results of Austin and Kerr–Li in the non-virtually-cyclic setting, this shows that two Bernoulli actions of any non-locally-finite countably infinite amenable group are Shannon orbit equivalent if and only if they are measure conjugate. We also show, at the opposite end of the stochastic spectrum, that every -odometer is Shannon orbit equivalent to the universal -odometer.

mathematics, applied
Signed Rényi Entropy and Quantum Second Laws

Adam Brandenburger,Pierfrancesco La Mura

2024-10-21

Abstract:We modify the Rényi (1961) axioms for entropy to apply to negative (``signed") measures as arise, for example, in phase-space representations of quantum mechanics. We obtain two new measures of (lack of) information about a system -- which we propose as signed analogs to classical Shannon entropy and classical Rényi entropy, respectively. We show that signed Rényi entropy witnesses non-classicality of a system. Specifically, a measure has at least one negative component if and only if signed Rényi $\alpha$-entropy is negative for some $\alpha > 1$. The corresponding non-classicality test does not work with signed Shannon entropy. We next show that signed Rényi $2k$-entropy, when $k$ is a positive integer, is Schur-concave. (An example shows that signed Shannon entropy is not Schur-concave.) We then establish an abstract quantum H-theorem for signed measures. We prove that signed Rényi $2k$-entropy is non-decreasing under classical (``decohering") evolution of a signed measure, where the latter could be a Wigner function or other phase-space representation of a quantum system. (An example shows that signed Shannon entropy may be non-monotonic.) We also provide a characterization of the Second Law for signed Rényi $2$-entropy in terms of what we call eventual classicalization of evolution of a system. We conclude with an argument that signed Rényi $2$-entropy of the Wigner function is constant under Moyal bracket evolution.

Quantum Physics,Mathematical Physics
Renyi extrapolation of Shannon entropy

Karol Zyczkowski

DOI: https://doi.org/10.48550/arXiv.quant-ph/0305062

2003-05-12

Quantum Physics

Abstract:Relations between Shannon entropy and Renyi entropies of integer order are discussed. For any N-point discrete probability distribution for which the Renyi entropies of order two and three are known, we provide an lower and an upper bound for the Shannon entropy. The average of both bounds provide an explicit extrapolation for this quantity. These results imply relations between the von Neumann entropy of a mixed quantum state, its linear entropy and traces.
A note on Bianchi-Donà's proof to the variance formula of von Neumann entropy

Lu Wei

DOI: https://doi.org/10.48550/arXiv.1906.10303

2019-06-25

Information Theory

Abstract:Bianchi and Don\`{a} [1] have recently reported a proof to the variance formula of von Neumann entropy, which was conjectured in [2] and firstly proved in [3]. The purpose of this short note is to show that, despite having a different starting point, the subsequent calculations (omitted in [1]) leading to the result are essentially the same as in [3].
Variability as a better characterization of Shannon entropy

Gabriele Carcassi,Christine A Aidala,Julian Barbour

DOI: https://doi.org/10.1088/1361-6404/abe361

2021-05-19

European Journal of Physics

Abstract:Abstract The Shannon entropy, one of the cornerstones of information theory, is widely used in physics, particularly in statistical mechanics. Yet its characterization and connection to physics remain vague, leaving ample room for misconceptions and misunderstanding. We will show that the Shannon entropy can be fully understood as measuring the variability of the elements within a given distribution: it characterizes how much variation can be found within a collection of objects. We will see that it is the only indicator that is continuous and linear, that it quantifies the number of yes/no questions (i.e. bits) that are needed to identify an element within the distribution, and we will see how applying this concept to statistical mechanics in different ways leads to the Boltzmann, Gibbs and von Neumann entropies.

physics, multidisciplinary,education, scientific disciplines
A remark on entropy numbers

Vladimir Temlyakov

DOI: https://doi.org/10.48550/arXiv.2008.13030

2020-08-30

Abstract:Talagrand's fundamental result on the entropy numbers is slightly improved. Our proof uses different ideas based on results from greedy approximation.

Numerical Analysis,Functional Analysis

A note on Shannon entropy

Shannon's entropy revisited

Generalized Shannon-Khinchin Axioms and Uniqueness Theorem for Pseudo-additive Entropies

A generalization of Costa's Entropy Power Inequality

Axioms and uniqueness theorem for Tsallis entropy

Generalization of Shannon-Khinchin Axioms to Nonextensive Systems and the Uniqueness Theorem

Tsallis entropy and generalized Shannon additivity

Shannon's entropy and its bounds for some a priori known equiprobable states

Equivalence of additive-combinatorial linear inequalities for Shannon entropy and differential entropy

Comment on "Stronger subadditivity of entropy" by Lieb and Seiringer <a class="link-https" data-arxiv-id="math-ph/0412009" href="https://arxiv.org/abs/math-ph/0412009">math-ph/0412009</a>

A characterization of von Neumann entropy using functors

Nonsymmetric entropy I: basic concepts and results

On a recent conjecture by Z. Van Herstraeten and N.J. Cerf for the quantum Wigner entropy

A unified formulation of entropy and its application

A Characterization of Entropy as a Universal Monoidal Natural Transformation

Entropy, virtual Abelianness and Shannon orbit equivalence

Signed Rényi Entropy and Quantum Second Laws

Renyi extrapolation of Shannon entropy

A note on Bianchi-Donà's proof to the variance formula of von Neumann entropy

Variability as a better characterization of Shannon entropy

A remark on entropy numbers