Abstract:As the amount of the digital data grows explosively, Data deduplication has gained increasing attention for its space-efficient functionality that not only reduces the storage space requirement by eliminating duplicate data but also minimizes the transmission of redundant data in data-intensive storage systems. Most existing state-ofthe-art deduplication methods remove redundant data at either the file level or the chunk level (e.g. Fixed-Sized Chunking and Content-Defined Chunking). But the four stages of the traditional data deduplication process, chunking, fingerprinting, indexing, writing the metadata & unique data chunks, are time consuming in storage systems, especially the processes of chunking and fingerprinting take up significant CPU resources. Since the computing power of single-core stagnates while the throughput of storage devices continues to increase steadily (e.g. flash and PCM), the chunking and fingerprinting stages of deduplication are becoming much slower than the writing stage in real-world deduplication based storage systems. More specifically, the Rabin-based chunking algorithm and the SHA-1or MD5-based fingerprinting algorithms all need to compute the hash digest, which may lengthen the write process to an unacceptable level for the required write speed in high performance storage systems. Currently, there are two general approaches to accelerating the time-consuming hash calculation and alleviating the computing bottleneck of data deduplication, namely, software-based and hardware-based methods. The former refers to employing a dedicated co-processor to minimize the time overheads of computing the hash function so that the deduplication-induced storage performance degradation becomes negligible or acceptable [1, 2, 4]. A good example of the hardware-based methods is called StoreGPU [5] that makes full use of the computing power of the GPU device to meet the computational demand of the hash calculation in storage systems. The software-based approaches exploit the parallelism of data deduplication instead of employing faster computing devices. Liu et al. [5] and Guo et al. [3] have attempted to improve the write performance by pipelining the Fixed-Sized Chunking (FSC) process. Because of the internal content dependency, it remains a challenge to fully exploit the parallelism in the chunking and fingerprinting tasks of the Content-Defined Chunking (CDC) based deduplication approaches. In this report, we propose P-Dedupe, a deduplication system for high performance data storage that pipelines and parallelizes the compute-intensive deduplication processes to remove the write bottleneck. PDedupe exploits the pipelining among the deduplication data units (e.g. chunks and files) and parallelism among the deduplication functional units (e.g. the fingerprinting and chunking tasks) by making full use of the idle computing resources in a multicoreor manycore-based computer system. P-Dedupe aims to remove the time overheads of hashing and shift the deduplication bottleneck from the CPU to the IO, so as to easily embed data deduplication into a normal data storage system with little or no impact on the write performance.

GLE-Dedup: A Globally–Locally Even Deduplication by Request-Aware Placement for Better Read Performance

Even Data Placement For Load Balance In Reliable Distributed Deduplication Storage Systems

A Delayed Container Organization Approach to Improve Restore Speed for Deduplication Systems.

DA Placement: A Dual-Aware Data Placement in a Deduplicated and Erasure-Coded Storage System

Speed-Dedup: A New Deduplication Framework for Enhanced Performance and Reduced Overhead in Scale-Out Storage

GreDedup: A Greedy-Based Application-Aware Data Routing Strategy for Distributed Deduplication

Try Managing Your Deduplication Fine-Grained-ly: A Multi-tiered and Dynamic SLA-Driven Deduplication Framework for Primary Storage.

From Hyper-dimensional Structures to Linear Structures: Maintaining Deduplicated Data’s Locality

Application-Aware Big Data Deduplication in Cloud Environment

A Novel Optimization Method to Improve De-duplication Storage System Performance

The Doctrine of MEAN: Realizing Deduplication Storage at Unreliable Edge.

The Dilemma Between Deduplication and Locality: Can Both Be Achieved?

DedupHR: Exploiting Content Locality to Alleviate Read/Write Interference in Deduplication-based Flash Storage

POD: Performance Oriented I/O Deduplication for Primary Storage Systems in the Cloud

Building a High-performance Fine-grained Deduplication Framework for Backup Storage with High Deduplication Ratio

Exploiting the Data Redundancy Locality to Improve the Performance of Deduplication-Based Storage Systems

Optimizing the Performance of Consistency-Aware Deduplication Using Persistent Memory

Optimizing the Restoration Performance of Deduplication Systems Through an Energy-Saving Data Layout

Leveraging Data Deduplication to Improve the Performance of Primary Storage Systems in the Cloud

Accelerating Data Deduplication by Exploiting Pipelining and Parallelism with Multicore or Manycore Processors

Accelerating Content-Defined-chunking Based Data Deduplication by Exploiting Parallelism.