Abstract:Images occupy a large amount of storage in data centers. To cope with the explosive growth of the image storage requirement, image compression techniques are devised to shrink the size of every single image at first. Furthermore, image deduplication methods are proposed to reduce the storage cost as they could be used to eliminate redundancy among images. However, state-of-the-art image deduplication methods either can only eliminate file-level coarse-grained redundancy or cannot guarantee lossless deduplication. In this work, we propose a new lossless image deduplication framework to eliminate fine-grained redundancy among images. It first decodes images to expose similarity, then eliminates fine-grained redundancy on the decoded data by delta compres-sion, and finally re-compresses the remaining data by image compression encoding. Based on this framework, we propose a novel lossless similarity-based deduplication (SBD) scheme for decoded image data (called imDedup). Specifically, imDedup uses a novel and fast sampling method (called Feature Map) to detect similar images in a two-dimensional way, which greatly reduces computation overhead. Meanwhile, it uses a novel delta encoder (called Idelta) which incorporates image compression encoding characteristics into deduplication to guarantee the remaining deduplicated image data to be friendly re-compressed via image encoding, which significantly improves the compression ratio. We implement a prototype of imDedup for JPEG images, and demonstrate its superiority on four datasets: Compared with exact image deduplication, imDedup achieves a 19%-38% higher compression ratio by efficiently eliminating fine-grained redundancy. Compared with the similarity detector and delta encoder of state-of-the-art SBD schemes running on the decoded image data, imDedup achieves a 1.8×-3.4× higher throughput and a 1.3 ×-1. 6 × higher compression ratio, respectively.

Lossless Compression of Time Series Data with Generalized Deduplication

DEC: An Efficient Deduplication-Enhanced Compression Approach

Ddelta: A Deduplication-Inspired Fast Delta Compression Approach

Machete: an Efficient Lossy Floating-Point Compressor Designed for Time Series Databases

Data Deduplication with Random Substitutions

The Design of a Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy for JPEG Image Storage Systems

Erasing-based lossless compression method for streaming floating-point time series

Lossless Data Compression for Time-Series Sensor Data Based on Dynamic Bit Packing

A Lossless Electrocardiogram Compression System Based on Dual-Mode Prediction and Error Modeling.

Near Lossless Time Series Data Compression Methods using Statistics and Deviation

Reducing Data Fragmentation in Data Deduplication Systems via Partial Repetition and Coding

Imdedup: A Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy among Images

Spatiotemporally adaptive compression for scientific dataset with feature preservation -- a case study on simulation data with extreme climate events analysis

Performance Optimization for Relative-Error-Bounded Lossy Compression on Scientific Data.

Real-Time Lossless Compression for Ultrahigh-Density Synchrophasor and Point-on-Wave Data

A Comprehensive Study of the Past, Present, and Future of Data Deduplication

Use cases of lossy compression for floating-point data in scientific data sets

Accelerating Content-Defined-chunking Based Data Deduplication by Exploiting Parallelism.

Time-universal data compression and prediction

Elf: Erasing-based Lossless Floating-Point Compression.

Learned Compression of Nonlinear Time Series With Random Access