Abstract:The era of big data has been constantly producing a huge amount of small files that will cause problems such as uneven distribution of copies and massive random readings and writings, resulting in delays in data accessions and degrading performances. By aiming at resolving the problems of an object storage system concerning the organization of small file objects such as layout imbalance, uneven loading, and adaptability of poor data migration, this article introduces a new strategy for object organization called object multi-tiered balanced organization strategy (OMBOS) with improved performance on massive small files in a real object storage system. Apart from the conventional strategy, the OMBOS achieves a multi-level balanced object by establishing weight indicators that reflect the comprehensive performance of nodes, including the balanced distribution of objects between replica sets and the balanced distribution of I/O requests in the replica sets as well as adapting to the equilibrium of the scale changes of the system dynamics. The OMBOS establishes a comprehensive performance evaluation model based on the fuzzy analytic hierarchy process and calculates the performance weights of the nodes of the object storages and replica sets. Then, the consistent hash-based object placement between the replica sets algorithm (BROP) and the I/O-loading balance in the replica set algorithm (IRIB) are utilized to implement performance-weighted objects in the object storage system through a tiered organization when the performance weights are taken into account. When the system scale changes flexibly, data migration is the only performed process in the nodes by adding or removing nodes, so data will not be migrated between existing nodes. Hence, the amount of data migration reaches the theoretical setting. Experiments show that the OMBOS achieves a balanced layout of objects based on performance weights, which meets the balancing and adaptability requirements of data organization. When compared to the original layout strategy through the balanced layout of objects, it performs nearly double the random access method.

An Efficient Storage Technology for Object-Based Distributed Storage System

An Efficient Big Data Storage Service Architecture

A Cache Design of Load Balancing System for Object-Based Storage

Design and Implementation of Load Balancing Based on Ceph Object Storage Cluster

File System Implementation Method to Improve the Network Storage Performance

A High-Speed Heterogeneous Cluster File System for Network-Attached Storage

Research and Design of MDS in Distributed Storage System

A Design of High Scalable Metadata Cluster in Mass Storage System

Optimization and performance improvement of distributed data storage in hybrid storage systems

A Self-Organizing Distributed Memory Cache For Data Sharing Applications In Cluster Environment

Optimizing File Transfer in Distributed Storage System Based on Reconfiguration of Storage Structure

WPAR: A Weight-Based Metadata Management Strategy for Petabyte-Scale Object Storage Systems

DWC2: A dynamic weight-based cooperative caching scheme for object-based storage cluster

A Modern Service-Oriented Distributed Storage Solution

Optimizing Object Storage System by the Object Multi-Tiered Balanced Organization

Research on Metadata Management Scheme of Distributed File System

An efficient metadata management method in large distributed storage systems

Distributed Heterogeneous Storage Based On Data Value

Optimization of Mass Storage Systems by Quality of Service

Content-aware network storage system supporting metadata retrieval

Dynamic Load Balance Algorithm for Big-data Distributed Storage