Abstract:We consider the lossless compression bound of any individual data sequence. If we fit the data by a parametric model, the entropy quantity $nH({\hat \theta}_n)$ obtained by plugging in the maximum likelihood estimate is an underestimate of the bound, where $n$ is the number of words. Shtarkov showed that the normalized maximum likelihood (NML) distribution or code length is optimal in a minimax sense for any parametric family. We show by the local asymptotic normality that the NML code length for the exponential families is $nH(\hat \theta_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \int_{\Theta} |I(\theta)|^{1/2}\, d\theta+o(1)$, where $d$ is the model dimension or dictionary size, and $|I(\theta)|$ is the determinant of the Fisher information matrix. We also demonstrate that sequentially predicting the optimal code length for the next word via a Bayesian mechanism leads to the mixture code, whose pathwise length is given by $nH({\hat \theta}_n) +\frac{d}{2}\log \, \frac{n}{2\pi} +\log \frac{|\, I({\hat \theta}_n)|^{1/2}}{w({\hat \theta}_n)}+o(1) $, where $w(\theta)$ is a prior. The asymptotics apply to not only discrete symbols but also continuous data if the code length for the former is replaced by the description length for the latter. The analytical result is exemplified by calculating compression bounds of protein-encoding DNA sequences under different parsing models. Typically, the highest compression is achieved when the parsing is in phase of the amino acid codons. On the other hand, the compression rates of pseudo-random sequences are larger than 1 regardless parsing models. These model-based results are in consistency with that random sequences are incompressible as asserted by the Kolmogorov complexity theory. The empirical lossless compression bound is particularly more accurate when dictionary size is relatively large.

X-ray studies of the phases and phase transitions of liquid crystals

Capacity of Frequency-based Channels: Encoding Information in Molecular Concentrations

Information Theory of Composite Sequence Motifs: Mutational and Biophysical Determinants of Complex Molecular Recognition

Physical complexity of symbolic sequences

Set-based complexity and biological information

Information-theoretic View of Sequence Organization in a Genome

Thermodynamic Capacity of a Protein

Information capacity of a network of spiking neurons

Beyond Countable Alphabets: an Extension of the Information-Spectrum Approach.

Information Theory of Genomes

Capacity Results for the Noisy Shuffling Channel

Capacity for Electromagnetic Information Theory

Subsystem Information Capacity in Random Circuits and Hamiltonian Dynamics

The information-carrying capacity of certain quantum channels

Capacity Definitions for General Channels with Receiver Side Information

On the Capacity of DNA Labeling

Empirical Lossless Compression Bound of a Data Sequence

The Capacity of the Weighted Read Channel

A formulation of the channel capacity of biochemical signaling cascades

Information theory in molecular biology

The Channel Capacity of Channelrhodopsin and Other Intensity-Driven Signal Transduction Receptors