A Multi-faceted Analysis of the Performance Variability of Virtual Machines

Luciano Baresi,Tommaso Dolci,Giovanni Quattrocchi,Nicholas Rasi

2023-09-21

Abstract:Cloud computing and virtualization solutions allow one to rent the virtual machines (VMs) needed to run applications on a pay-per-use basis, but rented VMs do not offer any guarantee on their performance. Cloud platforms are known to be affected by performance variability, but a better understanding is still required. This paper moves in that direction and presents an in-depth, multi-faceted study on the performance variability of VMs. Unlike previous studies, our assessment covers a wide range of factors: 16 VM types from 4 well-known cloud providers, 10 benchmarks, and 28 different metrics. We present four new contributions. First, we introduce a new benchmark suite (VMBS) that let researchers and practitioners systematically collect a diverse set of performance data. Second, we present a new indicator, called Variability Indicator, that allows for measuring variability in the performance of VMs. Third, we illustrate an analysis of the collected data across four different dimensions: resources, isolation, time, and cost. Fourth, we present multiple predictive models based on Machine Learning that aim to forecast future performance and detect time patterns. Our experiments provide important insights on the resource variability of VMs, highlighting differences and similarities between various cloud providers. To the best of our knowledge, this is the widest analysis ever conducted on the topic.

Software Engineering,Distributed, Parallel, and Cluster Computing

What problem does this paper attempt to address?

This paper attempts to address the problem of performance variability of virtual machines (VMs) in cloud computing environments. Specifically, the authors point out that although cloud computing and virtualization solutions allow users to rent the virtual machines required to run applications on a pay - as - you - go basis, these rented virtual machines do not provide any performance guarantees. Cloud platforms are known to be affected by performance variability, but the understanding of this phenomenon is still insufficient. Therefore, this paper aims to conduct an in - depth study of the variability of virtual machine performance through multi - aspect analysis.

The main contributions of the paper include:
1. **Developed a new benchmark test suite (VMBS)**: This tool enables researchers and practitioners to systematically collect various performance data.
2. **Proposed a new metric - Variability Indicator (VI)**: This metric is used to measure the variability of virtual machine performance.
3. **Conducted multi - dimensional analysis of the collected data**: Analyzed from four dimensions of resources, isolation, time, and cost to better characterize variability.
4. **Developed machine - learning - based prediction models**: These models are used to predict future performance and detect time patterns.

Through these methods, the paper provides important insights into the variability of virtual machine resources and reveals the differences and similarities between different cloud service providers. The research results are helpful for creating performance models and deriving quality indicators, thereby better understanding and predicting the performance of cloud platforms.