Abstract:The existing resemblance detection approaches fail to identify similar chunks with arbitrary similarity by setting a similarity threshold. In this work, the authors propose a novel resemblance detection scheme called Chunk2vec, which uses a deep neural network, Sentence‐BERT, to extract an approximate feature vector for each chunk while preserving its similarity with other chunks and applies the Approximate Nearest Neighbour Search technique to find the chunk's fingerprint feature vector with any given similarity range. This novel approach can significantly improve the accuracy of resemblance detection for post‐deduplication delta compression and greatly reduces the network transmission overhead between servers and clients. Delta compression, as a complementary technique for data deduplication, has gained widespread attention in network storage systems. It can eliminate redundant data between non‐duplicate but similar chunks that cannot be identified by data deduplication. The network transmission overhead between servers and clients can be greatly reduced by using data deduplication and delta compression techniques. Resemblance detection is a technique that identifies similar chunks for post‐deduplication delta compression in network storage systems. The existing resemblance detection approaches fail to detect similar chunks with arbitrary similarity by setting a similarity threshold, which can be suboptimal. In this paper, the authors propose Chunk2vec, a resemblance detection scheme for delta compression that utilizes deep learning techniques and Approximate Nearest Neighbour Search technique to detect similar chunks with any given similarity range. Chunk2vec uses a deep neural network, Sentence‐BERT, to extract an approximate feature vector for each chunk while preserving its similarity with other chunks. The experimental results on five real‐world datasets indicate that Chunk2vec improves the accuracy of resemblance detection for delta compression and achieves higher compression ratio than the state‐of‐the‐art resemblance detection technique.

Finesse: Fine-Grained Feature Locality Based Fast Resemblance Detection For Post-Deduplication Delta Compression

Exploiting Multiple Similarity Spaces for Efficient and Flexible Incremental Update of Mobile Apps

A Delayed Container Organization Approach to Improve Restore Speed for Deduplication Systems.

Chunk Content is not Enough: Chunk-Context Aware Resemblance Detection for Deduplication Delta Compression

Ultrasonic Data Compression Based on Feature Extraction and Delta Encoding

The Design of a Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy for JPEG Image Storage Systems

Applying Delta Compression to Packed Datasets for Efficient Data Reduction

Chunk2vec: A novel resemblance detection scheme based on Sentence‐BERT for post‐deduplication delta compression in network transmission

Fine-search for image copy detection based on local affine-invariant descriptor and spatial dependent matching

FuzzyDedup: Secure Fuzzy Deduplication for Cloud Storage

Improving Restore Performance for In-Line Backup System Combining Deduplication and Delta Compression

MODIFIED KERNEL-BASED NONLINEAR FEATURE EXTRACTION

Towards Efficient Network Compression Via Few-Shot Slimming.

A feature compression method based on similarity matching

Joint Rate-Distortion Optimization For Simultaneous Texture And Deep Feature Compression Of Facial Images

A Data Structure for Efficient File Deduplication in Cloud Storage

A Novel Optimization Method to Improve De-duplication Storage System Performance

AFEC: Adaptive Feature Extraction Modules for Learned Image Compression

DAQE: Enhancing the Quality of Compressed Images by Exploiting the Inherent Characteristic of Defocus

Deep CNN based online image deduplication technique for cloud storage system

Toward Intelligent Sensing: Intermediate Deep Feature Compression