Abstract:Erasure coding is a storage-efficient alternative to replication for achieving reliable data backup in distributed storage systems. During the storage process, traditional erasure codes require a unique source node to create and upload all the redundant data to the different storage nodes. However, such a source node may have limited communication and computation capabilities, which constrain the storage process throughput. Moreover, the source node and the different storage nodes might not be able to send and receive data simultaneously -- e.g., nodes might be busy in a datacenter setting, or simply be offline in a peer-to-peer setting -- which can further threaten the efficacy of the overall storage process. In this paper we propose an "in-network" redundancy generation process which distributes the data insertion load among the source and storage nodes by allowing the storage nodes to generate new redundant data by exchanging partial information among themselves, improving the throughput of the storage process. The process is carried out asynchronously, utilizing spare bandwidth and computing resources from the storage nodes. The proposed approach leverages on the local repairability property of newly proposed erasure codes tailor made for the needs of distributed storage systems. We analytically show that the performance of this technique relies on an efficient usage of the spare node resources, and we derive a set of scheduling algorithms to maximize the same. We experimentally show, using availability traces from real peer-to-peer applications as well as Google data center availability and workload traces, that our algorithms can, depending on the environment characteristics, increase the throughput of the storage process significantly (up to 90% in data centers, and 60% in peer-to-peer settings) with respect to the classical naive data insertion approach.

Model and Evaluation of Redundant Data Organization for High Availability in Structured Overlay Networks

Study on Data Redundancy Scheme in Kademlia Cloud Storage System

Stationary and Adaptive Replication Approach to Data Availability in Structured Peer-to-peer Overlay Networks.

Redundancy-Reduction-Based Hierarchical Design in Synchronization of Multi-Agent Systems

Reliability of Wireless Sensor Networks Based on Redundancy of Cluster-Heads

Cascading Failure Model of Interdependent Power Networks Based on Load Redistribution

Reliability Assessment of Stochastic Networks with ER Connectivity and ER Dependency

Reliability Analysis of Distributed Storage Systems Considering Data Loss and Theft

Topology-Aware Node Selection for Data Regeneration in Heterogeneous Distributed Storage Systems

In-Network Redundancy Generation for Opportunistic Speedup of Backup

Smart Ring: A Model of Node Failure Detection in High Available Cloud Data Center.

Design and Reliability Analysis of a Novel Redundancy Topology Architecture

Hybrid Overlay Structure Based on Random Walks

Heterogeneity-aware Data Regeneration in Distributed Storage Systems

The Optimization of Replica Distribution in the Unstructured Overlays

Accelerating Data Regeneration for Distributed Storage Systems with Heterogeneous Link Capacities

Redundancy Maintenance in P2P Storage Networks

Comparing Maintenance Strategies for Overlays

Availability Analysis of Redundant and Replicated Cloud Services with Bayesian Networks

Dependability modeling and analysis for the virtual clusters

Design and Implementation of One Redundancy Function in Wireless Tree Networks