Abstract:Erasure coding is a storage-efficient alternative to replication for achieving reliable data backup in distributed storage systems. During the storage process, traditional erasure codes require a unique source node to create and upload all the redundant data to the different storage nodes. However, such a source node may have limited communication and computation capabilities, which constrain the storage process throughput. Moreover, the source node and the different storage nodes might not be able to send and receive data simultaneously -- e.g., nodes might be busy in a datacenter setting, or simply be offline in a peer-to-peer setting -- which can further threaten the efficacy of the overall storage process. In this paper we propose an "in-network" redundancy generation process which distributes the data insertion load among the source and storage nodes by allowing the storage nodes to generate new redundant data by exchanging partial information among themselves, improving the throughput of the storage process. The process is carried out asynchronously, utilizing spare bandwidth and computing resources from the storage nodes. The proposed approach leverages on the local repairability property of newly proposed erasure codes tailor made for the needs of distributed storage systems. We analytically show that the performance of this technique relies on an efficient usage of the spare node resources, and we derive a set of scheduling algorithms to maximize the same. We experimentally show, using availability traces from real peer-to-peer applications as well as Google data center availability and workload traces, that our algorithms can, depending on the environment characteristics, increase the throughput of the storage process significantly (up to 90% in data centers, and 60% in peer-to-peer settings) with respect to the classical naive data insertion approach.

Fast Recovery Techniques for Erasure-coded Clusters in Non-uniform Traffic Network.

Multi-level Forwarding and Scheduling Recovery Algorithm in Rapidly-changing Network for Erasure-coded Clusters

Repair Tree: Fast Repair for Single Failure in Erasure-Coded Distributed Storage Systems

Tree-structured Parallel Regeneration for Multiple Data Losses in Distributed Storage Systems Based on Erasure Codes

Edge Computing-Empowered Large-Scale Traffic Data Recovery Leveraging Low-Rank Theory

Parallelized In-Network Aggregation for Failure Repair in Erasure-Coded Storage Systems

Towards In-network Acceleration of Erasure Coding

NetEC: Accelerating Erasure Coding Reconstruction with In-Network Aggregation

Rewiring 2 Links is Enough: Accelerating Failure Recovery in Production Data Center Networks.

Recovery from Link Failures in Networks with Arbitrary Topology via Diversity Coding

Fast Connection Recovery Against Region Failures with Landmark-Based Source Routing

Network Content Parallel Recovery System Based on Multi-core Processor

STORE: Data recovery with approximate minimum network bandwidth and disk I/O in distributed storage systems

In-Network Redundancy Generation for Opportunistic Speedup of Backup

A Quick Recovery Strategy for Network Failure under Multipath Multicast Transmissions

Bandwidth-Aware Node Repair Optimization for Distributed Storage System Based on Simple Regenerating Code

A Solution to the Network Challenges of Data Recovery in Erasure-coded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster

Deep Reinforcement Learning-Based Network Routing Technology for Data Recovery in Exa-Scale Cloud Distributed Clustering Systems

HTPC: heterogeneous traffic-aware partition coding for random packet spraying in data center networks

Accelerating Data Regeneration for Distributed Storage Systems with Heterogeneous Link Capacities

Reconsidering Single Disk Failure Recovery for Erasure Coded Storage Systems: Optimizing Load Balancing in Stack-Level