Abstract:Performance variability, stemming from nondeterministic hardware and software behaviors or deterministic behaviors such as measurement bias, is a well-known phenomenon of computer systems which increases the difficulty of comparing computer performance metrics and is slated to become even more of a concern as interest in Big Data analytic increases. Conventional methods use various measures (such as geometric mean) to quantify the performance of different benchmarks to compare computers without considering this variability which may lead to wrong conclusions. In this paper, we propose three resampling methods for performance evaluation and comparison: a randomization test for a general performance comparison between two computers, bootstrapping confidence estimation, and an empirical distribution and five-number-summary for performance evaluation. The results show that for both PARSEC and high-variance BigDataBench benchmarks 1) the randomization test substantially improves our chance to identify the difference between performance comparisons when the difference is not large; 2) bootstrapping confidence estimation provides an accurate confidence interval for the performance comparison measure (e.g., ratio of geometric means); and 3) when the difference is very small, a single test is often not enough to reveal the nature of the computer performance due to the variability of computer systems.We further propose using empirical distribution to evaluate computer performance and a five-number-summary to summarize computer performance. We use published SPEC 2006 results to investigate the sources of performance variation by predicting performance and relative variation for 8,236 machines. We achieve a correlation of predicted performances of 0.992 and a correlation of predicted and measured relative variation of 0.5. Finally, we propose the utilization of a novel biplotting technique to visualize the effectiveness of benchmarks and cluster machines by behavior. We illustrate the results and conclusion through detailed Monte Carlo simulation studies and real examples.

Benchmarking optimization software with performance profiles

Nested Performance Profiles for Benchmarking Software

From Standardized Data Formats to Standardized Tools for Optimization Algorithm Benchmarking

Using Affine Combinations of BBOB Problems for Performance Assessment

Benchmarking in Optimization: Best Practice and Open Issues

Towards a Benchmarking Suite for Kernel Tuners

Computer Comparisons in the Presence of Performance Variation

Object Database Benchmarks

Speedoo: prioritizing performance optimization opportunities.

A Sample Profile-Based Optimization Method with Better Precision

Paradox-Free Analysis for Comparing the Performance of Optimization Algorithms

Generating Performance Distributions Via Probabilistic Symbolic Execution.

A Visual Web Tool to Perform What-If Analysis of Optimization Approaches

IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics

COCO: Performance Assessment

MO-IOHinspector: Anytime Benchmarking of Multi-Objective Algorithms using IOHprofiler

A Case for Dataset Specific Profiling

Profiling Based Optimization Method for CPU-GPU Heterogeneous Parallel Processing System

Branch-and-Bound Performance Estimation Programming: A Unified Methodology for Constructing Optimal Optimization Methods

Benchmarking Algorithms for Submodular Optimization Problems Using IOHProfiler

The Impact of Profiling Versus Static Analysis in Precision Tuning