Abstract:We consider the lossless compression bound of any individual data sequence. If we fit the data by a parametric model, the entropy quantity $nH({\hat \theta}_n)$ obtained by plugging in the maximum likelihood estimate is an underestimate of the bound, where $n$ is the number of words. Shtarkov showed that the normalized maximum likelihood (NML) distribution or code length is optimal in a minimax sense for any parametric family. We show by the local asymptotic normality that the NML code length for the exponential families is $nH(\hat \theta_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \int_{\Theta} |I(\theta)|^{1/2}\, d\theta+o(1)$, where $d$ is the model dimension or dictionary size, and $|I(\theta)|$ is the determinant of the Fisher information matrix. We also demonstrate that sequentially predicting the optimal code length for the next word via a Bayesian mechanism leads to the mixture code, whose pathwise length is given by $nH({\hat \theta}_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \frac{|\, I({\hat \theta}_n)|^{1/2}}{w({\hat \theta}_n)}+o(1) $, where $w(\theta)$ is a prior. The asymptotics apply to not only discrete symbols but also continuous data if the code length for the former is replaced by the description length for the latter. The analytical result is exemplified by calculating compression bounds of protein-encoding DNA sequences under different parsing models. Typically, the highest compression is achieved when the parsing is in phase of the amino acid codons. On the other hand, the compression rates of pseudo-random sequences are larger than 1 regardless parsing models. These model-based results are in consistency with that random sequences are incompressible as asserted by the Kolmogorov complexity theory. The empirical lossless compression bound is particularly more accurate when dictionary size is relatively large.

Universality of Logarithmic Loss in Lossy Compression

Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff

Lossy Compression of Individual Sequences Revisited: Fundamental Limits of Finite-State Encoders

Minimum Entropy Coupling with Bottleneck

Information Theoretic Lower Bounds on Negative Log Likelihood

Empirical Lossless Compression Bound of a Data Sequence

Optimally Controllable Perceptual Lossy Compression

A Family of LZ78-based Universal Sequential Probability Assignments

Joint Data and Semantics Lossy Compression: Nonasymptotic Converse Bounds and Second-Order Asymptotics

Low-Rate, Low-Distortion Compression with Wasserstein Distortion

A Framework for Robust Lossy Compression of Heavy-Tailed Sources

On Distributed Lossy Coding of Symmetrically Correlated Gaussian Sources

The Lossy Common Information of Correlated Sources

On statistical learning via the lens of compression

Lossy Computing with Side Information Via Multi-Hypergraphs.

On the rate-distortion-perception-semantics tradeoff in low-rate regime for lossy compression

Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets

Local Decoding in Distributed Compression

Importance Matching Lemma for Lossy Compression with Side Information

A Distributionally Robust Approach to Shannon Limits using the Wasserstein Distance