Abstract:Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test must achieve a reasonable trade-off between result quality and testing time. This becomes particularly challenging in Java context, where the software undergoes a warm-up phase of execution, due to just-in-time compilation. During this phase, performance measurements are subject to severe fluctuations, which may adversely affect quality of performance test results. However, these approaches often provide suboptimal estimates of the warm-up phase, resulting in either insufficient or excessive warm-up iterations, which may degrade result quality or increase testing time. There is still a lack of consensus on how to properly address this problem. Here, we propose and study an AI-based framework to dynamically halt warm-up iterations at runtime. Specifically, our framework leverages recent advances in AI for Time Series Classification (TSC) to predict the end of the warm-up phase during test execution. We conduct experiments by training three different TSC models on half a million of measurement segments obtained from JMH microbenchmark executions. We find that our framework significantly improves the accuracy of the warm-up estimates provided by state-of-practice and state-of-the-art methods. This higher estimation accuracy results in a net improvement in either result quality or testing time for up to +35.3% of the microbenchmarks. Our study highlights that integrating AI to dynamically estimate the end of the warm-up phase can enhance the cost-effectiveness of Java performance testing.

TFPaaS : Test-first Performance as a Service to Cloud for Software Testing Environment

Automated Model-Based Performance Testing for PaaS Cloud Services

LTF: A Model-Based Load Testing Framework for Web Applications

P4SC: A High Performance and Flexible Framework for Service Function Chain

Testing As a Service over Cloud

A Novel Framework to Carry out Cloud Penetration Test

A cloud-based TaaS infrastructure with tools for SaaS validation, performance and scalability evaluation

A Framework of Testing As a Service

Vtestkit: A Performance Benchmarking Framework for Virtualization Environments

A Configurable Web Service Performance Testing Framework

[email protected]: the virtual test lab on the cloud

Vee@Cloud: the virtual test lab on the cloud

Testing as a Service (TaaS) on Clouds

TaPS: A Performance Evaluation Suite for Task-based Execution Frameworks

AI-driven Java Performance Testing: Balancing Result Quality with Testing Time

Enabling Performance Intelligence for Application Adaptation in the Future Internet.

Cost-Performance Driven Resource Configuration for Database Applications in IaaS Cloud Environments

Efficiently Detecting Performance Changes in FaaS Application Releases

A performance optimization framework for hardware/software co design

Towards virtualized and automated software performance test architecture

Embedding Performance Testing in Agile Software Model