Abstract:We consider the lossless compression bound of any individual data sequence. If we fit the data by a parametric model, the entropy quantity $nH({\hat \theta}_n)$ obtained by plugging in the maximum likelihood estimate is an underestimate of the bound, where $n$ is the number of words. Shtarkov showed that the normalized maximum likelihood (NML) distribution or code length is optimal in a minimax sense for any parametric family. We show by the local asymptotic normality that the NML code length for the exponential families is $nH(\hat \theta_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \int_{\Theta} |I(\theta)|^{1/2}\, d\theta+o(1)$, where $d$ is the model dimension or dictionary size, and $|I(\theta)|$ is the determinant of the Fisher information matrix. We also demonstrate that sequentially predicting the optimal code length for the next word via a Bayesian mechanism leads to the mixture code, whose pathwise length is given by $nH({\hat \theta}_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \frac{|\, I({\hat \theta}_n)|^{1/2}}{w({\hat \theta}_n)}+o(1) $, where $w(\theta)$ is a prior. The asymptotics apply to not only discrete symbols but also continuous data if the code length for the former is replaced by the description length for the latter. The analytical result is exemplified by calculating compression bounds of protein-encoding DNA sequences under different parsing models. Typically, the highest compression is achieved when the parsing is in phase of the amino acid codons. On the other hand, the compression rates of pseudo-random sequences are larger than 1 regardless parsing models. These model-based results are in consistency with that random sequences are incompressible as asserted by the Kolmogorov complexity theory. The empirical lossless compression bound is particularly more accurate when dictionary size is relatively large.

Joint Data and Semantics Lossy Compression: Nonasymptotic Converse Bounds and Second-Order Asymptotics

Indirect Lossy Source Coding with Observed Source Reconstruction: Nonasymptotic Bounds and Second-Order Asymptotics

Information-Theoretic Limits on Compression of Semantic Information

Semantic Compression with Side Information: A Rate-Distortion Perspective

On Distributed Lossy Coding of Symmetrically Correlated Gaussian Sources

On the Second-Order Achievabilities of Indirect Quadratic Lossy Source Coding

Joint Source-Channel Secrecy Using Hybrid Coding

Joint Task and Data Oriented Semantic Communications: A Deep Separate Source-channel Coding Scheme

Asymptotically Optimal Joint Sampling and Compression for Timely Status Updates: Age-Distortion Tradeoff

Lossy Computing with Side Information Via Multi-Hypergraphs.

Learning Scalable ℓ∞-Constrained Near-lossless Image Compression Via Joint Lossy Image and Residual Compression

A Practical Approach to Lossy Joint Source-Channel Coding

Lossy Compression of Individual Sequences Revisited: Fundamental Limits of Finite-State Encoders

Discriminatory Lossy Source Coding: Side Information Privacy

Learning Scalable l∞-constrained Near-lossless Image Compression via Joint Lossy Image and Residual Compression

Excess Distortion Exponent Analysis for Semantic-Aware MIMO Communication Systems

Output-Constrained Lossy Source Coding With Application to Rate-Distortion-Perception Theory

Empirical Lossless Compression Bound of a Data Sequence