Abstract:Erasure codes have been considered as one of the most promising techniques for data reliability enhancement and storage efficiency in modern distributed storage systems. However, erasure codes often suffer from a time-consuming coding process which makes them nearly impractical. The opportunity to solve this problem probably rely on the parallelization of erasure-code-based application on the modern multi/many-core processors to fully take advantage of the adequate hardware resources on those platforms. However, the complicated data allocation and limited I/O throughput pose a great challenge on the parallelization. To address this challenge, we propose a general multi-threaded parallel coding approach in this work. The approach consists of a general multi-threaded parallel coding model named as MTPerasure, and two detailed parallel coding algorithms, named as sdaParallel and ddaParallel, respectively, adapting to different I/O circumstances. MTPerasure is a general parallel coding model focusing on the high level data allocation, and it is applicable for all erasure codes and can be implemented without any modifications of the low level coding algorithms. The sdaParallel divides the data into several parts and the data parts are allocated to different threads statically in order to eliminate synchronization latency among multiple threads, which improves the parallel coding performance under the dummy I/O mode. The ddaParallel employs two threads to execute the I/O reading and writing on the basis of small pieces independently, which increases the I/O throughput. Furthermore, the data pieces are assigned to the coding thread dynamically. A special thread scheduling algorithm is also proposed to reduce thread migration latency. To evaluate our proposal, we parallelize the popular open source library jerasure based on our approach. And a detailed performance comparison with the original sequential coding program indicates that the proposed parallel approach outperforms the original sequential program by an extraordinary speedups from 1.4x up to 7x, and achieves better utilization of the computation and I/O resources.

Highly Scalable Parallel Arithmetic Coding on Multi-Core Processors Using LDPC Codes

A scheme of parallel arithmetic coding.

Massively LDPC Decoding on Multicore Architectures

A Novel Decoding Algorithm for Parallel Concatenation of LDPC Codes

Highly Parallel Line-Based Image Coding for Many Cores

A pure software ldpc decoder on a multi-core processor platform with reduced inter-processor communication cost

Comparison of Parallelization Strategies for Min-Sum Decoding of Irregular LDPC Codes

Design of a Multi-Core Scheduling Scheme for Tera-bit/s LDPC Decoding.

Code Compression Approach for Heterogeneous Dual-Core SOC Based on Arithmetic Coding

An Efficient Parallel Coding Scheme In Erasure-Coded Storage Systems

Adaptive-distributed Arithmetic Coding for Lossless Compression

High-Efficient Parallel Cavlc Encoders on Heterogeneous Multicore Architectures

Rate-Compatible And High-Throughput Architecture Designs For Encoding Ldpc Codes

Constructing of LDPC Code with High Speed Parallel Decoding Structure

Systematic Construction, Verification and Implementation Methodology for Ldpc Codes

An efficient multi-rate LDPC-CC decoder with layered decoding algorithm

Multi-Gb/s LDPC Code Design and Implementation

Design and Efficient Hardware Implementation Schemes for Non-Quasi-cyclic LDPC Codes

A Parallel Arithmetic Array for Accelerating Compute-Intensive Applications.

Implementation aspects of LDPC convolutional codes

High-Throughput LDPC-CC Decoders Based on Storage, Arithmetic, and Control Improvements