Abstract:The advent of the Big Data era has brought considerable challenges to storing and managing massive data. Moreover, distributed storage systems are critical to the pressure and storage capacity costs. The Ceph cloud storage system only selects data storage nodes based on node storage capacity. This node selection method results in load imbalance and limited storage scenarios in heterogeneous storage systems. Therefore, we add node heterogeneity, network state, and node load as performance weights to the CRUSH algorithm and optimize the performance of the Ceph system by improving load balancing. We designed a cloud storage system model based on Software Defined Network (SDN) technology. This system model can avoid the tedious configuration and significant measurement overhead required to obtain network status in traditional network architecture. Then we propose adaptive read and write optimization algorithms based on SDN technology. The Object Storage Device (OSD) is initially classified based on the Node Heterogeneous Resource Classification Strategy. Then the SDN technology is used to obtain network and load conditions in real-time and an OSD performance prediction model is built to obtain weights for performance impact factors. Finally, a mathematical model is proposed for multi-attribute decision making in conjunction with the OSD state and its prediction model. Furthermore, this model is addressed to optimize read and write performance adaptively. Compared with the original Ceph system, TOPSIS_PA improves the performance of reading operations by 36%; TOPSIS_CW and TOPSIS_PACW algorithms improve the elastic read performance by 23 to 60% and 36 to 85%, and the elastic write performance by 180 to 468% and 188 to 611%, respectively.

Topology-Aware Efficient Storage Scheme for Fault-Tolerant Storage Systems in Data Centers

A Practical Cross-Datacenter Fault-Tolerance Algorithm in the Cloud Storage System.

Data Placement Strategy in Data Center Distributed Storage Systems

Fault Tolerant Storage and Data Access Optimization in Data Center Networks.

A Comprehensive Repair Scheme for Distributed Storage Systems

A Novel XOR-based Fault-Tolerance Mechanism for Distributed Storage System

Storage Technologies for Big Data

NADE: Nodes Performance Awareness and Accurate Distance Evaluation for Degraded Read in Heterogeneous Distributed Erasure Code-Based Storage.

BCDC: A High-Performance, Server-Centric Data Center Network

Toward A Scalable, Fault-Tolerant, High-Performance Optical Data Center Architecture.

A Novel Network Structure with Power Efficiency and High Availability for Data Centers.

Bandwidth-Aware Node Repair Optimization for Distributed Storage System Based on Simple Regenerating Code

An adaptive read/write optimized algorithm for Ceph heterogeneous systems via performance prediction and multi-attribute decision making

Disaster Protection in Inter-DataCenter Networks leveraging Cooperative Storage.

NCCloud: A Network-Coding-Based Storage System in a Cloud-of-Clouds

Constructing Multiple CISTs on BCube-Based Data Center Networks in the Occurrence of Switch Failures

Design And Performance Analysis Of Distributed Fault Tolerant Storage Systems

The CORE Storage Primitive: Cross-Object Redundancy for Efficient Data Repair & Access in Erasure Coded Storage

Design and Optimization for Distributed Indexing Scheme in Switch-Centric Cloud Storage System

Accelerating Data Regeneration for Distributed Storage Systems with Heterogeneous Link Capacities

A Data Layout and Fast Failure Recovery Scheme for Distributed Storage Systems with Mixed Erasure Codes