Abstract:We consider the lossless compression bound of any individual data sequence. If we fit the data by a parametric model, the entropy quantity $nH({\hat \theta}_n)$ obtained by plugging in the maximum likelihood estimate is an underestimate of the bound, where $n$ is the number of words. Shtarkov showed that the normalized maximum likelihood (NML) distribution or code length is optimal in a minimax sense for any parametric family. We show by the local asymptotic normality that the NML code length for the exponential families is $nH(\hat \theta_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \int_{\Theta} |I(\theta)|^{1/2}\, d\theta+o(1)$, where $d$ is the model dimension or dictionary size, and $|I(\theta)|$ is the determinant of the Fisher information matrix. We also demonstrate that sequentially predicting the optimal code length for the next word via a Bayesian mechanism leads to the mixture code, whose pathwise length is given by $nH({\hat \theta}_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \frac{|\, I({\hat \theta}_n)|^{1/2}}{w({\hat \theta}_n)}+o(1) $, where $w(\theta)$ is a prior. The asymptotics apply to not only discrete symbols but also continuous data if the code length for the former is replaced by the description length for the latter. The analytical result is exemplified by calculating compression bounds of protein-encoding DNA sequences under different parsing models. Typically, the highest compression is achieved when the parsing is in phase of the amino acid codons. On the other hand, the compression rates of pseudo-random sequences are larger than 1 regardless parsing models. These model-based results are in consistency with that random sequences are incompressible as asserted by the Kolmogorov complexity theory. The empirical lossless compression bound is particularly more accurate when dictionary size is relatively large.

Computing the optimal error exponential function for fixed-length lossy coding in discrete memoryless sources

Computation of the optimal error exponent function for fixed-length lossy source coding in discrete memoryless sources

Optimum Overflow Thresholds in Variable-Length Source Coding Allowing Non-Vanishing Error Probability

Robust Coding for Lossy Computing with Receiver-Side Observation Costs

Optimum Trade-offs Between the Error Exponent and the Excess-Rate Exponent of Variable-Rate Slepian-Wolf Coding

Mismatched Decoding: Error Exponents, Second-Order Rates and Saddlepoint Approximations

Output-Constrained Lossy Source Coding With Application to Rate-Distortion-Perception Theory

Distributed Source Coding Using Constrained-Random-Number Generators

Lossy Computing with Side Information Via Multi-Hypergraphs.

Exact Asymptotics for the Random Coding Error Probability

A Framework for Robust Lossy Compression of Heavy-Tailed Sources

Fixed-Length Error Resilient Code and Its Application in Video Coding

Optimum end-to-end distortion estimation for error resilient video coding

On Distributed Lossy Coding of Symmetrically Correlated Gaussian Sources

Distributed Approximate Computing with Constant Locality

A New Finite-Horizon Dynamic Programming Analysis of Nonanticipative Rate-Distortion Function for Markov Sources

Empirical Lossless Compression Bound of a Data Sequence

Joint Source-Channel Rate-Distortion Optimization for H.264 Video Coding over Error-Prone Networks

Error Exponent for Nakagami-m Fading Massive MIMO Channels

Computation of Rate-Distortion-Perception Function under f-Divergence Perception Constraints

Rate-Distortion-Perception Tradeoff of Variable-Length Source Coding for General Information Sources