Abstract:Real-world large-scale applications expose more and more pressures to storage services of modern supercomputers. Supercomputers have been introducing new storage devices and technologies to meet the performance requirements of various applications, leading to more complicated architectures. High I/O demand of applications and the complicated and shared storage architectures make the issues, such as unbalanced load, I/O interference, system parameter configuration error, and node performance degradation, more frequently observed. And it is challenging to both achieve high I/O performance on application level and efficiently utilize scarce storage resources. We propose AIOT, an end-to-end and adaptive I/O optimization tool for HPC storage systems, which introduces effective I/O performance modeling and several active tuning strategies to improve both the I/O performance of applications and the utilization of storage resources. AIOT provides a global view of the whole storage system and searches for the optimal end-to-end I/O path through flow network modeling. Moreover, AIOT tunes system parameters across multiple layers of the storage system by using the automated identified application I/O behaviors and the instant status of the workload of storage system. We verified the effectiveness of AIOT for balancing I/O load, resolving I/O interference, improving I/O performance by configuring appropriate system parameters, and avoiding I/O performance degradation caused by abnormal nodes through quite a few real-world cases. AIOT has helped to save over ten millions of core-hours during the deployment on Sunway TaihuLight since July 2021. It's worth mentioning that our proposed AIOT is capable of managing other I/O optimization methods across various storage platforms.

Review of I/O Performance Optimization Technology for Big Data Storage System

Optimization Factor Analysis Of Large-Scale Join Queries On Different Platforms

High performance disk I/O system

A Self-adaptive and Prediction-based I/O Performance Optimization Method

I/O Path Based Performance Model for Storage Systems

Performance Analysis and Framework Optimization of Open Source Cloud Storage System

The Application of Big Data Analytics in Optimizing Logistics: a Developmental Perspective Review

Lessons Learned from Optimizing the Sunway Storage System for Higher Application I/O Performance

Study on Workload-aware Tuning Methods of Storage Subsystem

An End-to-end and Adaptive I/O Optimization Tool for Modern HPC Storage Systems

Review of Control Optimization for Automated Storage and Retrieval Systems

Design and application of new storage systems

A Novel Optimization Method to Improve De-duplication Storage System Performance

Study and performance analysis of IP-based storage system

Performance optimization of computing task scheduling based on the Hadoop big data platform

A Survey on I/O Virtualization and Optimization

A Survey of Index Structure in Big Data Era

In-Memory Big Data Management and Processing: A Survey

Optimization Control Strategies and Evaluation Metrics of Cooling Systems in Data Centers: A Review

Improving the Effective IO Throughput by Adaptive Read-Ahead Strategy for Private Cloud Storage Service

Design and Performance Analysis of Modern Computational Storage Devices: A Systematic Review