Abstract:As the cloud platform becomes a promising alternative to traditional HPC (high performance computing) centers or in-house clusters, the I/O bottleneck problem is highlighted in this new environment, typically with top-of-the-line compute instances but sub-par communication and I/O facilities. It has been observed that changing the cloud I/O system configurations, such as choices of file systems, number of I/O servers and their placement strategies, etc., will lead to a considerable variation in the performance and cost efficiency of I/O intensive parallel applications. However, storage system configuration is tedious and error-prone to do manually, even for expert users, leading to solutions that are grossly over-provisioned (low cost inefficiency), substantially under-performing (poor performance) or, in the worst case, both. This paper proposes ACIC, a system which automatically searches for optimized I/O system configurations from many candidates for each individual application running on a given cloud platform. ACIC takes advantage of machine learning models to perform performance/cost predictions. To tackle the high-dimensional parameter exploration space, we enable affordable, reusable, and incremental training on cloud platforms, guided by the Plackett and Burman Matrices for experiment design. Our evaluation results with four representative parallel applications indicate that ACIC consistently identifies optimal or near-optimal configurations among a large group of candidate settings. The top ACIC-recommended configuration is capable of improving the applications' performance by a factor of up to 10.5 (3.1 on average), and cost saving of up to 89 percent (51 percent on average), compared with a commonly used baseline I/O configuration. In addition, we carried out a small-scale user study for one of the test applications, which found that ACIC consistently beat the user and even the application's developer, often by a significant margin, in selecting optimized configurations.

ACIC: automatic cloud I/O configurator for parallel applications

Automatic Cloud I/O Configurator for I/O Intensive Parallel Applications

Acic: Automatic Cloud I/O Configurator For Hpc Applications

AI-oriented Workload Allocation for Cloud-Edge Computing.

One Optimized I/O Configuration Per HPC Application

An End-to-end and Adaptive I/O Optimization Tool for Modern HPC Storage Systems

CSC: Collaborative System Configuration for I/O-Intensive Applications in Multi-Tenant Clouds

Achelous: Enabling Programmability, Elasticity, and Reliability in Hyperscale Cloud Networks.

ACE: Towards Application-Centric Edge-Cloud Collaborative Intelligence

Weight-based strategy for an I/O-intensive application at a cloud data center.

Congestion integrated control in virtualized clouds

Optimizing I/O Performance Through Effective Vcpu Scheduling Interference Management

Automatic, Application-Aware I/O Forwarding Resource Allocation

Ultra-low Latency Cloud-Fog Computing for Industrial Internet of Things

Automatic Scaling of Internet Applications for Cloud Computing Services

An Energy-Efficient System on a Programmable Chip Platform for Cloud Applications.

IO QoS: A New Disk I/O Scheduler Module with QoS Guarantee for Cloud Platform

An Adaptive Approach to Better Load Balancing in a Consumer-Centric Cloud Environment

A New Approach to Double I/O Performance for Ceph Distributed File System in Cloud Computing

Accelerated Cloud for Artificial Intelligence (ACAI)

Performance Bottleneck Analysis and Resource Optimized Distribution Method for IoT Cloud Rendering Computing System in Cyber-Enabled Applications