Abstract:Data deduplication, one of the key features of modern Big Data storage devices, is the process of removing replicas of data chunks stored by different users. Despite the importance of deduplication, several drawbacks of the method, such as storage robustness and file fragmentation, have not been previously analyzed from a theoretical point of view. Storage robustness pertains to ensuring that deduplicated data can be used to reconstruct the original files without service disruptions and data loss. Fragmentation pertains to the problems of placing deduplicated data chunks of different user files in a proximity-preserving linear order, since neighboring chunks of the same file may be stored in sectors far apart on the server. This work proposes a new theoretical model for data fragmentation and introduces novel graph- and coding-theoretic approaches for reducing fragmentation via limited duplication (repetition coding) and coded deduplication (e.g., linear coding). In addition to alleviating issues with fragmentation, limited duplication and coded deduplication can also serve the dual purpose of increasing the robusteness of the system design. The contributions of our work are three-fold. First, we describe a new model for file structures in the form of self-avoiding (simple) paths in specialized graphs. Second, we introduce several new metrics for measuring the fragmentation level in deduplication systems on graph-structured files, including the stretch metric that captures the worst-case "spread" of adjacent data chunks within a file when deduplicated and placed on the server; and, the jump metric that captures the worst-case number of times during the reconstruction process of a file that one has to change the readout location on the server. For the stretch metric, we establish a connection between the level of fragmentation and the bandwidth of the file-graph. In particular, ...

Entropy Coding of Unordered Data Structures

Random Cycle Coding: Lossless Compression of Cluster Assignments via Bits-Back Coding

Random Permutation Codes: Lossless Source Coding of Non-Sequential Data

Random Edge Coding: One-Shot Bits-Back Coding of Large Labeled Graphs

Efficient Coding and Decoding Algorithm Based on Generalized Huffman Tree

Overcoming the compression limit of the individualsequence (zero order empirical entropy) using the Set Shaping Theory

Rotating labeling of entropy coders for synthetic DNA data storage

Low-Complexity Vector Source Coding for Discrete Long Sequences with Unknown Distributions

A constrained Shannon-Fano entropy coder for image storage in synthetic DNA

A Practical Concatenated Coding Scheme for Noisy Shuffling Channels with Coset-based Indexing

Rate-limited Shuffling for Distributed Computing

Compressing Sets and Multisets of Sequences

On Random Tree Structures, Their Entropy, and Compression

Optimal Coding for Randomized Kolmogorov Complexity and Its Applications

Coding for Random Projections

Pyramid Codes: Flexible Schemes to Trade Space for Access Efficiency in Reliable Data Storage Systems

Capacity Results for the Noisy Shuffling Channel

Shuffle and Learn: Minimizing Mutual Information for Unsupervised Hashing

Reducing Data Fragmentation in Data Deduplication Systems via Partial Repetition and Coding

Where the patterns are: repetition-aware compression for colored de Bruijn graphs

Universal Lossless Compression of Graphical Data