Abstract:Data deduplication, one of the key features of modern Big Data storage devices, is the process of removing replicas of data chunks stored by different users. Despite the importance of deduplication, several drawbacks of the method, such as storage robustness and file fragmentation, have not been previously analyzed from a theoretical point of view. Storage robustness pertains to ensuring that deduplicated data can be used to reconstruct the original files without service disruptions and data loss. Fragmentation pertains to the problems of placing deduplicated data chunks of different user files in a proximity-preserving linear order, since neighboring chunks of the same file may be stored in sectors far apart on the server. This work proposes a new theoretical model for data fragmentation and introduces novel graph- and coding-theoretic approaches for reducing fragmentation via limited duplication (repetition coding) and coded deduplication (e.g., linear coding). In addition to alleviating issues with fragmentation, limited duplication and coded deduplication can also serve the dual purpose of increasing the robusteness of the system design. The contributions of our work are three-fold. First, we describe a new model for file structures in the form of self-avoiding (simple) paths in specialized graphs. Second, we introduce several new metrics for measuring the fragmentation level in deduplication systems on graph-structured files, including the stretch metric that captures the worst-case "spread" of adjacent data chunks within a file when deduplicated and placed on the server; and, the jump metric that captures the worst-case number of times during the reconstruction process of a file that one has to change the readout location on the server. For the stretch metric, we establish a connection between the level of fragmentation and the bandwidth of the file-graph. In particular, ...

Data Deduplication Techniques for Big Data Storage Systems

A Delayed Container Organization Approach to Improve Restore Speed for Deduplication Systems.

Decentralized and Privacy Sensitive Data De-Duplication Framework for Convenient Big Data Management in Cloud Backup Systems

Emerging Research Trends in Data Deduplication: A Bibliometric Analysis from 2010 to 2023

Droplet: A Distributed Solution of Data Deduplication

Reducing Data Fragmentation in Data Deduplication Systems via Partial Repetition and Coding

Redundancy elimination in IoT oriented big data: a survey, schemes, open challenges and future applications

SAUD: Semantics-Aware and Utility-Driven Deduplication Framework for Primary Storage.

A Novel Optimization Method to Improve De-duplication Storage System Performance

D3: A Dynamic Dual-Phase Deduplication Framework for Distributed Primary Storage.

A secure framework for managing data in cloud storage using rapid asymmetric maximum based dynamic size chunking and fuzzy logic for deduplication

A Thorough Investigation of Content-Defined Chunking Algorithms for Data Deduplication

Boafft: Distributed Deduplication for Big Data Storage in the Cloud

Secure Batch Deduplication without Dual Servers in Backup System

SecDedoop: Secure Deduplication with Access Control of Big Data in the HDFS/Hadoop Environment

Semantic data de-duplication for archival storage systems

Efficient Hybrid Inline and Out-of-line Deduplication for Backup Storage

ADMAD: Application-Driven Metadata Aware De-duplication Archival Storage System

Battling the Digital Forensic Backlog through Data Deduplication

A Remote Data Backup System with Deduplication

Speed-Dedup: A New Deduplication Framework for Enhanced Performance and Reduced Overhead in Scale-Out Storage