Abstract:Performance variability, stemming from nondeterministic hardware and software behaviors or deterministic behaviors such as measurement bias, is a well-known phenomenon of computer systems which increases the difficulty of comparing computer performance metrics and is slated to become even more of a concern as interest in Big Data analytic increases. Conventional methods use various measures (such as geometric mean) to quantify the performance of different benchmarks to compare computers without considering this variability which may lead to wrong conclusions. In this paper, we propose three resampling methods for performance evaluation and comparison: a randomization test for a general performance comparison between two computers, bootstrapping confidence estimation, and an empirical distribution and five-number-summary for performance evaluation. The results show that for both PARSEC and high-variance BigDataBench benchmarks 1) the randomization test substantially improves our chance to identify the difference between performance comparisons when the difference is not large; 2) bootstrapping confidence estimation provides an accurate confidence interval for the performance comparison measure (e.g., ratio of geometric means); and 3) when the difference is very small, a single test is often not enough to reveal the nature of the computer performance due to the variability of computer systems.We further propose using empirical distribution to evaluate computer performance and a five-number-summary to summarize computer performance. We use published SPEC 2006 results to investigate the sources of performance variation by predicting performance and relative variation for 8,236 machines. We achieve a correlation of predicted performances of 0.992 and a correlation of predicted and measured relative variation of 0.5. Finally, we propose the utilization of a novel biplotting technique to visualize the effectiveness of benchmarks and cluster machines by behavior. We illustrate the results and conclusion through detailed Monte Carlo simulation studies and real examples.

How NOT to Fool the Masses When Giving Performance Results for Quantum Computers

Defining Standard Strategies for Quantum Benchmarks

Benchmarking quantum computers

SoK: Benchmarking the Performance of a Quantum Computer

Better than classical? The subtle art of benchmarking quantum machine learning models

Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach

Twelve Ways to Fool the Masses When Giving Parallel-in-Time Results

Towards Robust Benchmarking of Quantum Optimization Algorithms

Experimental Benchmarking of an Automated Deterministic Error-Suppression Workflow for Quantum Algorithms

Comparisons among the Performances of Randomized-framed Benchmarking Protocols under T1, T2 and Coherent Error Models

Quantum Algorithm Exploration using Application-Oriented Performance Benchmarks

Classically Spoofing System Linear Cross Entropy Score Benchmarking

Benchmarking the performance of quantum computing software

Re-examining the quantum volume test: Ideal distributions, compiler optimizations, confidence intervals, and scalable resource estimations

A methodology for comparing and benchmarking quantum devices

Performance Benchmarking of Quantum Algorithms for Hard Combinatorial Optimization Problems: A Comparative Study of non-FTQC Approaches

Scalable Full-Stack Benchmarks for Quantum Computers

Optimization Applications as Quantum Performance Benchmarks

Seeking Quantum Speedup Through Spin Glasses: the Good, the Bad, and the Ugly

Computer Comparisons in the Presence of Performance Variation

Volumetric Benchmarking of Quantum Computing Noise Models