Abstract:Recently, two extensions of Wyner's common information\textemdash exact and Rényi common informations\textemdash were introduced respectively by Kumar, Li, and El Gamal (KLE), and the present authors. The class of common information problems involves determining the minimum rate of the common input to two independent processors needed to exactly or approximately generate a target joint distribution. For the exact common information problem, exact generation of the target distribution is required, while for Wyner's and $\alpha$-Rényi common informations, the relative entropy and Rényi divergence with order $\alpha$ were respectively used to quantify the discrepancy between the synthesized and target distributions. The exact common information is larger than or equal to Wyner's common information. However, it was hitherto unknown whether the former is strictly larger than the latter for some joint distributions. In this paper, we first establish the equivalence between the exact and $\infty$-Rényi common informations, and then provide single-letter upper and lower bounds for these two quantities. For doubly symmetric binary sources, we show that the upper and lower bounds coincide, which implies that for such sources, the exact and $\infty$-Rényi common informations are completely characterized. Interestingly, we observe that for such sources, these two common informations are strictly larger than Wyner's. This answers an open problem posed by KLE. Furthermore, we extend Wyner's, $\infty$-Rényi, and exact common informations to sources with countably infinite or continuous alphabets, including Gaussian sources.

Linear Convergence in Hilbert's Projective Metric for Computing Augustin Information and a Rényi Information Measure

On the moment convergence rates of LIL in Hilbert space

Computing Augustin Information via Hybrid Geodesically Convex Optimization

Algorithms for Computing the Augustin--Csiszár Mutual Information and Lapidoth--Pfister Mutual Information

The Convergence of the Sums of Independent Random Variables under the Sub-Linear Expectations

Proximal Point Algorithms on Hadamard Manifolds: Linear Convergence and Finite Termination

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

Hilbert's projective metric for functions of bounded growth and exponential convergence of Sinkhorn's algorithm

Convergence Analysis of a Method for Variational Inclusions

Generalized Convergence Rates Results for Linear Inverse Problems in Hilbert Spaces

On Statistical Measure Theory

On Exact and $\infty$-Rényi Common Informations

Proving Information Inequalities by Gaussian Elimination

Robust and Fast Measure of Information Via Low-Rank Representation

Measure theory of statistical convergence

Mutual Information, Information-Theoretic Thresholds and the Condensation Phenomenon at Positive Temperature

Linear Information Coupling Problems

Sibson's $α$-Mutual Information and its Variational Representations

Rates of Fisher information convergence in the central limit theorem for nonlinear statistics