Abstract:The Shannon entropy of a random variable X has much behaviour analogous to a signed measure. Previous work has explored this connection by defining a signed measure on abstract sets, which are taken to represent the information that different random variables contain. This construction is sufficient to derive many measure-theoretical counterparts to information quantities such as the mutual information $I(X; Y) = \mu(\tilde{X} \cap \tilde{Y})$, the joint entropy $H(X,Y) = \mu(\tilde{X} \cup \tilde{Y})$, and the conditional entropy $H(X|Y) = \mu(\tilde{X} \setminus \tilde{Y})$. Here we provide concrete characterisations of these abstract sets and a corresponding signed measure, and in doing so we demonstrate that there exists a much finer decomposition with intuitive properties which we call the logarithmic decomposition (LD). We show that this signed measure space has the useful property that its logarithmic atoms are easily characterised with negative or positive entropy, while also being consistent with Yeung's I-measure. We present the usability of our approach by re-examining the Gács-Körner common information and the Wyner common information from this new geometric perspective and characterising it in terms of our logarithmic atoms - a property we call logarithmic decomposability. We present possible extensions of this construction to continuous probability distributions before discussing implications for quality-led information theory. Lastly, we apply our new decomposition to examine the Dyadic and Triadic systems of James and Crutchfield and show that, in contrast to the I-measure alone, our decomposition is able to qualitatively distinguish between them.

Expressions for the Entropy of Binomial-Type Distributions

Logarithmic and power-law entropies from convexity

Self-Normalized Moderate Deviation and Laws of the Iterated Logarithm under G-Expectation

Precise Rates in the Law of the Logarithm for Negatively Associated Random Variables

The Law of Logarithm for Arrays of Random Variables under Sub-linear Expectations

Relative Entropy via Distribution of Observables

Entropy Formula for Endomorphisms: Relations Between Entropy, Exponents and Dimension

Properties of the Shannon, Rényi and other entropies: dependence in parameters, robustness in distributions and extremes

Entropies of the Poisson distribution as functions of intensity: "normal" and "anomalous" behavior

Linear Combination of Order Statistics Moments from Log-Extended Exponential Geometric Distribution with Applications to Entropy

Entropies for complex systems: generalized-generalized entropies

Limits of sums for binomial and Eulerian numbers and their associated distributions

Origins of the Combinatorial Basis of Entropy

Balanced distribution-energy inequalities and related entropy bounds

Entropy Formula for Random Dynamical Systems: Relations Between Entropy, Exponents and Dimension

Explicit expressions for joint moments of $n$-dimensional elliptical distributions

Log-concavity, ultra-log-concavity, and a maximum entropy property of discrete compound Poisson measures

A Logarithmic Decomposition and a Signed Measure Space for Entropy

Bernoulli sums and Rényi entropy inequalities

Entropy of Compact Operators with Applications to Landau-Pollak-Slepian Theory and Sobolev Spaces

Logical Entropy and Negative Probabilities in Quantum Mechanics