Abstract:Recent technological advancements have led to the generation of huge amounts of data over the web, such as text, image, audio and video. Most of this data is high dimensional and sparse, for e.g., the bag-of-words representation used for representing text. Often, an efficient search for similar data points needs to be performed in many applications like clustering, nearest neighbour search, ranking and indexing. Even though there have been significant increases in computational power, a simple brute-force similarity-search on such datasets is inefficient and at times impossible. Thus, it is desirable to get a compressed representation which preserves the similarity between data points. In this work, we consider the data points as sets and use Jaccard similarity as the similarity measure. Compression techniques are generally evaluated on the following parameters --1) Randomness required for compression, 2) Time required for compression, 3) Dimension of the data after compression, and 4) Space required to store the compressed data. Ideally, the compressed representation of the data should be such, that the similarity between each pair of data points is preserved, while keeping the time and the randomness required for compression as low as possible. We show that the compression technique suggested by Pratap and Kulkarni also works well for Jaccard similarity. We present a theoretical proof of the same and complement it with rigorous experimentations on synthetic as well as real-world datasets. We also compare our results with the state-of-the-art "min-wise independent permutation", and show that our compression algorithm achieves almost equal accuracy while significantly reducing the compression time and the randomness.

SortComp (Sort-and-compress) - Towards a Universal Lossless Compression Scheme for Matrix and Tabular Data

Corra: Correlation-Aware Column Compression

Reordering Rows for Better Compression: Beyond the Lexicographic Order

LeCo: Lightweight Compression Via Learning Serial Correlations

Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets

Czip: A Fast Lossless Compression Algorithm for Climate Data

Regression Cubes with Lossless Compression and Aggregation

Efficient Compression Technique for Sparse Sets

Binary Matrices for Compressed Sensing.

Lightweight Correlation-Aware Table Compression

Enhancing Lossy Compression Through Cross-Field Information for Scientific Applications

Universal Graph Compression: Stochastic Block Models

A Prediction-Traversal Approach for Compressing Scientific Data on Unstructured Meshes with Bounded Error

A Survey on Error-Bounded Lossy Compression for Scientific Datasets

An Optimized Error-controlled MPI Collective Framework Integrated with Lossy Compression

A Novel Lossless Encoding Algorithm for Data Compression - Genomics Data as an Exemplar

Value-Compressed Sparse Column (VCSC): Sparse Matrix Storage for Redundant Data

Time-universal data compression and prediction

Data Compression using Rank-1 Lattices for Parameter Estimation in Machine Learning

A Versatile Compression Method for Floating-Point Data Stream