Abstract:Post-deduplication delta compression is a data reduction technique that calculates and stores the differences of the very similar but non-duplicate chunks in storage systems, which is able to achieve a very high compression ratio. However, the low throughput of widely-used resemblance detection approaches (e.g., N-Transform) usually becomes the bottleneck of delta compression systems due to introducing high computational overhead. Generally, this overhead mainly consists of two parts: ① calculating the rolling hash byte-by-byte across data chunks and ② applying multiple transforms on all the calculated rolling hash values. In this paper, we propose Odess, a fast and lightweight resemblance detection approach, that greatly reduces the computational overhead for resemblance detection while achieving high detection accuracy and high compression ratio. Specifically, Odess first utilizes a novel Subwindow-based Parallel Rolling hash method (called SWPR) using SIMD (i.e., Single Instruction Multiple Data [1]) to accelerate calculation of rolling hashes (corresponding to the first part of the overhead). Moreover, Odess uses a novel Content-Defined Sampling method to generate a much smaller proxy hash set from the whole rolling hash set, and then quickly applies transforms on this small hash set for resemblance detection (corresponding to the second part of the overhead). Evaluation results show that during the stage of resemblance detection, the Odess approach is ∼ 31.4 × and ∼ 7.9 × faster than the state-of-the-art N-Transform and Finesse (i.e., a recent variant of N-Transform [39]), respectively. When considering an end-to-end data reduction storage system, Odess-based system’s throughput is about 3.20 × and 1.41 × higher than N-Transform and Finesse-based systems’ throughput, respectively, while maintaining the high compression ratio of N-Transform and achieving ∼ 1.22 × higher compression ratio over Finesse.

Study of Homology Image Deduplication Technology Based on Haar Wavelet

Structural Similarity Measurement Based on Dual-Tree Complex Wavelet

The Design of a Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy for JPEG Image Storage Systems

Imdedup: A Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy among Images

Large-Scale Duplicate Detection for Web Image Search

Detecting Aligned Double JPEG Compression with the Same Quantization Matrix Based on Hamming Distance and Image Stability

Deep CNN based online image deduplication technique for cloud storage system

Fine-search for image copy detection based on local affine-invariant descriptor and spatial dependent matching

FuzzyDedup: Secure Fuzzy Deduplication for Cloud Storage

Haar-Like Wavelets on Hierarchical Trees

Double Trouble? Impact and Detection of Duplicates in Face Image Datasets

Study on Hybrid Compression Algorithm of CT Images

On Real-Time Detecting Duplicate Web Videos

Data De-duplication on Similar File Detection

A Fast Duplicate Chunk Identifying Method Based on Hierarchical Indexing Structure

The Design of Fast and Lightweight Resemblance Detection for Efficient Post-Deduplication Delta Compression

BitHash: an Efficient Bitwise Locality Sensitive Hashing Method with Applications

Double sliding window chunking algorithm for data deduplication in ocean observation

A Comprehensive Study of the Past, Present, and Future of Data Deduplication

Query Oriented Subspace Shifting for Near-Duplicate Image Detection

Locating topological structures in digital images via local homology