Abstract:The Shannon entropy of a random variable X has much behaviour analogous to a signed measure. Previous work has explored this connection by defining a signed measure on abstract sets, which are taken to represent the information that different random variables contain. This construction is sufficient to derive many measure-theoretical counterparts to information quantities such as the mutual information $I(X; Y) = \mu(\tilde{X} \cap \tilde{Y})$, the joint entropy $H(X,Y) = \mu(\tilde{X} \cup \tilde{Y})$, and the conditional entropy $H(X|Y) = \mu(\tilde{X} \setminus \tilde{Y})$. Here we provide concrete characterisations of these abstract sets and a corresponding signed measure, and in doing so we demonstrate that there exists a much finer decomposition with intuitive properties which we call the logarithmic decomposition (LD). We show that this signed measure space has the useful property that its logarithmic atoms are easily characterised with negative or positive entropy, while also being consistent with Yeung's I-measure. We present the usability of our approach by re-examining the Gács-Körner common information and the Wyner common information from this new geometric perspective and characterising it in terms of our logarithmic atoms - a property we call logarithmic decomposability. We present possible extensions of this construction to continuous probability distributions before discussing implications for quality-led information theory. Lastly, we apply our new decomposition to examine the Dyadic and Triadic systems of James and Crutchfield and show that, in contrast to the I-measure alone, our decomposition is able to qualitatively distinguish between them.

A Marginal Characterization of Entropy Functions for Conditional Mutually Independent Random Variables (with Application to Wyner's Common Information)

Conditional Entropy Formula with Respect to Monotonic Partitions

A Conditional Entropy Power Inequality for Dependent Variables

Conditional Deng Entropy, Joint Deng Entropy and Generalized Mutual Information

A Characterization of Entropy as a Universal Monoidal Natural Transformation

Conditional entropies and their relation to entanglement criteria

Entropy Functions and Determinant Inequalities

Entropic Uncertainty Relations for General Symmetric Informationally Complete Positive Operator-Valued Measures and Mutually Unbiased Measurements

Error Bounds on a Mixed Entropy Inequality

On two classes of Rényi entropy functions of a quantum channel

The subadditivity of quantum entropy in Gaussian quantum systems

A Local Characterization for Wyner Common Information

Sharp Continuity Bounds for Entropy and Conditional Entropy

Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem

Entropic Uncertainty Inequalities on Sparse Representation

Wyner's Common Information: Generalizations and A New Lossy Source Coding Interpretation

A Logarithmic Decomposition and a Signed Measure Space for Entropy

A New Extremal Entropy Inequality With Applications

A remark on conditional entropy

Generalised Entropies and Metric-Invariant Optimal Countermeasures for Information Leakage under Symmetric Constraints

An entropic uncertainty principle for mixed states