Abstract:As the cloud platform becomes a promising alternative to traditional HPC (high performance computing) centers or in-house clusters, the I/O bottleneck problem is highlighted in this new environment, typically with top-of-the-line compute instances but sub-par communication and I/O facilities. It has been observed that changing the cloud I/O system configurations, such as choices of file systems, number of I/O servers and their placement strategies, etc., will lead to a considerable variation in the performance and cost efficiency of I/O intensive parallel applications. However, storage system configuration is tedious and error-prone to do manually, even for expert users, leading to solutions that are grossly over-provisioned (low cost inefficiency), substantially under-performing (poor performance) or, in the worst case, both. This paper proposes ACIC, a system which automatically searches for optimized I/O system configurations from many candidates for each individual application running on a given cloud platform. ACIC takes advantage of machine learning models to perform performance/cost predictions. To tackle the high-dimensional parameter exploration space, we enable affordable, reusable, and incremental training on cloud platforms, guided by the Plackett and Burman Matrices for experiment design. Our evaluation results with four representative parallel applications indicate that ACIC consistently identifies optimal or near-optimal configurations among a large group of candidate settings. The top ACIC-recommended configuration is capable of improving the applications' performance by a factor of up to 10.5 (3.1 on average), and cost saving of up to 89 percent (51 percent on average), compared with a commonly used baseline I/O configuration. In addition, we carried out a small-scale user study for one of the test applications, which found that ACIC consistently beat the user and even the application's developer, often by a significant margin, in selecting optimized configurations.

An Autonomic Capacity Management Approach with Cloud Insight Towards Cost-Efficient Throughput Optimization for High Performance Computing

The Application of Autonomic Computing in High-Performance Server's Management System

Dynamic Resource Management in a HPC and Cloud Hybrid Environment

Autonomic Performance Management of Cloud Server Based on Adaptive Control Method

Efficiency-Aware Workload Optimizations of Heterogeneous Cloud Computing for Capacity Planning in Financial Industry

An Adaptive Approach to Better Load Balancing in a Consumer-Centric Cloud Environment

Energy-Efficient Scheduling of HPC Applications in Cloud Computing Environments

Acic: Automatic Cloud I/O Configurator For Hpc Applications

Resource Management in Sustainable Cyber-Physical Systems Using Heterogeneous Cloud Computing

Optimization Based Resource and Cooling Management for a High Performance Computing Data Center

Automatic Cloud I/O Configurator for I/O Intensive Parallel Applications

A Cost Saving and Load Balancing Task Scheduling Model for Computational Biology in Heterogeneous Cloud Datacenters.

Online Resource Management in Thermal and Energy Constrained Heterogeneous High Performance Computing

Power-Efficient Schemes Via Workload Characterization On The Intel'S Single-Chip Cloud Computer

Tuning Adaptive Computations for Performance Improvement of Autonomic Middleware in PaaS Cloud

A Runtime Available Resource Capacity Evaluation Model Based on the Concept of Similar Tasks.

CAWSAC: Cost-Aware Workload Scheduling and Admission Control for Distributed Cloud Data Centers.

An Adaptive Management Approach Collaborating Heterogeneous Resources in Cloud Service Environments

On Stochastic Performance and Cost-Aware Optimal Capacity Planning of Unreliable Infrastructure-as-a-Service Cloud.

A Speculative Approach to Spatial-Temporal Efficiency with Multi-Objective Optimization in a Heterogeneous Cloud Environment

Partial and Cost-Minimized Computation Offloading in Hybrid Edge and Cloud Systems