Abstract:As the cloud platform becomes a promising alternative to traditional HPC (high performance computing) centers or in-house clusters, the I/O bottleneck problem is highlighted in this new environment, typically with top-of-the-line compute instances but sub-par communication and I/O facilities. It has been observed that changing the cloud I/O system configurations, such as choices of file systems, number of I/O servers and their placement strategies, etc., will lead to a considerable variation in the performance and cost efficiency of I/O intensive parallel applications. However, storage system configuration is tedious and error-prone to do manually, even for expert users, leading to solutions that are grossly over-provisioned (low cost inefficiency), substantially under-performing (poor performance) or, in the worst case, both. This paper proposes ACIC, a system which automatically searches for optimized I/O system configurations from many candidates for each individual application running on a given cloud platform. ACIC takes advantage of machine learning models to perform performance/cost predictions. To tackle the high-dimensional parameter exploration space, we enable affordable, reusable, and incremental training on cloud platforms, guided by the Plackett and Burman Matrices for experiment design. Our evaluation results with four representative parallel applications indicate that ACIC consistently identifies optimal or near-optimal configurations among a large group of candidate settings. The top ACIC-recommended configuration is capable of improving the applications' performance by a factor of up to 10.5 (3.1 on average), and cost saving of up to 89 percent (51 percent on average), compared with a commonly used baseline I/O configuration. In addition, we carried out a small-scale user study for one of the test applications, which found that ACIC consistently beat the user and even the application's developer, often by a significant margin, in selecting optimized configurations.

iFast: Host-Side Logging for Scientific Applications

Streaming Data in HPC Workflows Using ADIOS

A High-Speed Heterogeneous Cluster File System for Network-Attached Storage

Exploring Scientific Application Performance Using Large Scale Object Storage

Imfssc: An In-Memory Distributed File System Framework For Super Computing

A Lightweight I/O Scheme to Facilitate Spatial and Temporal Queries of Scientific Data Analytics.

SciAP: A Programmable, High-Performance Platform for Large-Scale Scientific Data

ODDS: Optimizing Data-Locality Access for Scientific Data Analysis

HadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale Supercomputers

S4D-Cache: Smart Selective SSD Cache for Parallel I/O Systems

An End-to-end and Adaptive I/O Optimization Tool for Modern HPC Storage Systems

FASTDB: An Array Database System for Efficient Storing and Analyzing Massive Scientific Data.

Mechanisms of Client-Side Caching on High Performance Computers

Measurements and Analytics of Wide-Area File Transfers over Dedicated Connections

Automatic Cloud I/O Configurator for I/O Intensive Parallel Applications

Performance Measurements of Supercomputing and Cloud Storage Solutions

The Freeze-Frame File System

Aggregating Local Storage for Scalable Deep Learning I/O

Accelerating Large Scale Scientific Exploration through Data Diffusion

Experimental Analysis of File Transfer Rates over Wide-Area Dedicated Connections

A Data Optimizer for Region-Aware Self-describing Files in Scientific Computing