Abstract:As the cloud platform becomes a promising alternative to traditional HPC (high performance computing) centers or in-house clusters, the I/O bottleneck problem is highlighted in this new environment, typically with top-of-the-line compute instances but sub-par communication and I/O facilities. It has been observed that changing the cloud I/O system configurations, such as choices of file systems, number of I/O servers and their placement strategies, etc., will lead to a considerable variation in the performance and cost efficiency of I/O intensive parallel applications. However, storage system configuration is tedious and error-prone to do manually, even for expert users, leading to solutions that are grossly over-provisioned (low cost inefficiency), substantially under-performing (poor performance) or, in the worst case, both. This paper proposes ACIC, a system which automatically searches for optimized I/O system configurations from many candidates for each individual application running on a given cloud platform. ACIC takes advantage of machine learning models to perform performance/cost predictions. To tackle the high-dimensional parameter exploration space, we enable affordable, reusable, and incremental training on cloud platforms, guided by the Plackett and Burman Matrices for experiment design. Our evaluation results with four representative parallel applications indicate that ACIC consistently identifies optimal or near-optimal configurations among a large group of candidate settings. The top ACIC-recommended configuration is capable of improving the applications' performance by a factor of up to 10.5 (3.1 on average), and cost saving of up to 89 percent (51 percent on average), compared with a commonly used baseline I/O configuration. In addition, we carried out a small-scale user study for one of the test applications, which found that ACIC consistently beat the user and even the application's developer, often by a significant margin, in selecting optimized configurations.

Predictive modeling of I/O characteristics in high performance computing systems.

Prediction of High-Performance Computing Input/Output Variability and Its Application to Optimization for System Configurations

A Numerical Model Oriented Large-scale Parallel I/O Optimization Method.

Hybrid Performance Modeling And Analyzing Of Parallel Systems

Visual Analysis of Cloud Computing Performance Using Behavioral Lines

Analyzing & modeling the performance in Xen-based virtual cluster environment

A Taxonomy of Error Sources in HPC I/O Machine Learning Models

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

Precise Contention-Aware Performance Prediction on Virtualized Multicore System.

Performance and power modeling and prediction using MuMMI and 10 machine learning methods

I/O Burst Prediction for HPC Clusters using Darshan Logs

Predicting Output Performance of a Petascale Supercomputer

Automatic Cloud I/O Configurator for I/O Intensive Parallel Applications

Acic: Automatic Cloud I/O Configurator For Hpc Applications

Performance prediction of parallel computing models to analyze cloud-based big data applications

I/O-Conscious And Prediction-Enabled Virtual Machines Scheduling

Smart VM Co-Scheduling with the Precise Prediction of Performance Characteristics

Multi-node Power/Performance Modeling for HPC System

Estimation of the workload of a hybrid computing cluster when performing modeling tasks in materials science

Predicting System-level Power for a Hybrid Supercomputer

Performance and Power Modeling and Prediction Using MuMMI and Ten Machine Learning Methods