Abstract:We initiate the study of sub-linear sketching and streaming techniques for estimating the output size of common dictionary compressors such as Lempel-Ziv '77, the run-length Burrows-Wheeler transform, and grammar compression. To this end, we focus on a measure that has recently gained much attention in the information-theoretic community and which approximates up to a polylogarithmic multiplicative factor the output sizes of those compressors: the normalized substring complexity function $\delta$. We present a data sketch of $O(\epsilon^{-3}\log n + \epsilon^{-1}\log^2 n)$ words that allows computing a multiplicative $(1\pm \epsilon)$-approximation of $\delta$ with high probability, where $n$ is the string length. The sketches of two strings $S_1,S_2$ can be merged in $O(\epsilon^{-1}\log^2 n)$ time to yield the sketch of $\{S_1,S_2\}$, speeding up by orders of magnitude tasks such as the computation of all-pairs \emph{Normalized Compression Distances} (NCD). If random access is available on the input, our sketch can be updated in $O(\epsilon^{-1}\log^2 n)$ time for each character right-extension of the string. This yields a polylogarithmic-space algorithm for approximating $\delta$, improving exponentially over the working space of the state-of-the-art algorithms running in nearly-linear time. Motivated by the fact that random access is not always available on the input data, we then present a streaming algorithm computing our sketch in $O(\sqrt n \cdot \log n)$ working space and $O(\epsilon^{-1}\log^2 n)$ worst-case delay per character. We show that an implementation of our streaming algorithm can estimate {\delta} on a dataset of 189GB with a throughput of 203MB per minute while using only 5MB of RAM, and that our sketch speeds up the computation of all-pairs NCD distances by one order of magnitude, with applications to phylogenetic tree reconstruction.

The Compression method and applications

Exploiting Multiple Similarity Spaces for Efficient and Flexible Incremental Update of Mobile Apps

A Versatile Compression Method for Floating-Point Data Stream

Optimal (Euclidean) Metric Compression

Efficient Compression Technique for Sparse Sets

Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression

About some compression algorithms

Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor

Constrained Optimization with Compressed Gradients: A Dynamical Systems Perspective

Data Compression using Rank-1 Lattices for Parameter Estimation in Machine Learning

Sketching and Streaming for Dictionary Compression

Applications of Common Information to Computing Functions

Point Compression Schemes on Binary Edwards Curves

A Comprehensive Analysis on Image Encryption and Compression Techniques with the Assessment of Performance Evaluation Metrics

Optimal Distance Estimation Between Compressed Data Series.

On the Erdős distance problem

Using Low-Discrepancy Points for Data Compression in Machine Learning: An Experimental Comparison

Compressed Representations of Permutations, and Applications

Compressive mining: fast and optimal data mining in the compressed domain

Internet-oriented Optimization Schemes for Joint Compression and Encryption

The Minimal Compression Rate for Similarity Identification