Abstract:Shannon entropy is the shortest average codeword length a lossless compressor can achieve by encoding i.i.d. symbols. However, there are cases in which the objective is to minimize the \textit{exponential} average codeword length, i.e. when the cost of encoding/decoding scales exponentially with the length of codewords. The optimum is reached by all strategies that map each symbol $x_i$ generated with probability $p_i$ into a codeword of length $\ell^{(q)}_D(i)=-\log_D\frac{p_i^q}{\sum_{j=1}^Np_j^q}$. This leads to the minimum exponential average codeword length, which equals the Rényi, rather than Shannon, entropy of the source distribution. We generalize the established Arithmetic Coding (AC) compressor to this framework. We analytically show that our generalized algorithm provides an exponential average length which is arbitrarily close to the Rényi entropy, if the symbols to encode are i.i.d.. We then apply our algorithm to both simulated (i.i.d. generated) and real (a piece of Wikipedia text) datasets. While, as expected, we find that the application to i.i.d. data confirms our analytical results, we also find that, when applied to the real dataset (composed by highly correlated symbols), our algorithm is still able to significantly reduce the exponential average codeword length with respect to the classical `Shannonian' one. Moreover, we provide another justification of the use of the exponential average: namely, we show that by minimizing the exponential average length it is possible to minimize the probability that codewords exceed a certain threshold length. This relation relies on the connection between the exponential average and the cumulant generating function of the source distribution, which is in turn related to the probability of large deviations. We test and confirm our results again on both simulated and real datasets.

Entropy estimation of symbol sequences

Entropy and long-range correlations in random symbolic sequences

Determining the Number of Samples Required to Estimate Entropy in Natural Sequences

A Very Efficient Scheme for Estimating Entropy of Data Streams Using Compressed Counting

Dynamical complexity of short and noisy time series

Overcoming the compression limit of the individualsequence (zero order empirical entropy) using the Set Shaping Theory

On Nonlinear Complexity and Shannon's Entropy of Finite Length Random Sequences.

Empirical Lossless Compression Bound of a Data Sequence

Entropy Estimators for Markovian Sequences: A Comparative Analysis

An improved estimator of Shannon entropy with applications to systems with memory

Complexity and fractal dimensions for infinite sequences with positive entropy

Exponential Convergence Rate in Entropy

On the Eigenvalue and Shannon's Entropy of Finite Length Random Sequences

On nonlinear compression costs: when Shannon meets Rényi

Sublinear quantum algorithms for estimating von Neumann entropy

Computing the Entropy Rate of Information Source with Methods of Statistical Physics

Symbolic transfer entropy rate is equal to transfer entropy rate for bivariate finite-alphabet stationary ergodic Markov processes

Entropy numbers of finite-dimensional Lorentz space embeddings

Geometrical Interpretation of Shannon's Entropy Based on the Born Rule

Minimax rates of entropy estimation on large alphabets via best polynomial approximation