Abstract:Performance variability, stemming from nondeterministic hardware and software behaviors or deterministic behaviors such as measurement bias, is a well-known phenomenon of computer systems which increases the difficulty of comparing computer performance metrics and is slated to become even more of a concern as interest in Big Data analytic increases. Conventional methods use various measures (such as geometric mean) to quantify the performance of different benchmarks to compare computers without considering this variability which may lead to wrong conclusions. In this paper, we propose three resampling methods for performance evaluation and comparison: a randomization test for a general performance comparison between two computers, bootstrapping confidence estimation, and an empirical distribution and five-number-summary for performance evaluation. The results show that for both PARSEC and high-variance BigDataBench benchmarks 1) the randomization test substantially improves our chance to identify the difference between performance comparisons when the difference is not large; 2) bootstrapping confidence estimation provides an accurate confidence interval for the performance comparison measure (e.g., ratio of geometric means); and 3) when the difference is very small, a single test is often not enough to reveal the nature of the computer performance due to the variability of computer systems.We further propose using empirical distribution to evaluate computer performance and a five-number-summary to summarize computer performance. We use published SPEC 2006 results to investigate the sources of performance variation by predicting performance and relative variation for 8,236 machines. We achieve a correlation of predicted performances of 0.992 and a correlation of predicted and measured relative variation of 0.5. Finally, we propose the utilization of a novel biplotting technique to visualize the effectiveness of benchmarks and cluster machines by behavior. We illustrate the results and conclusion through detailed Monte Carlo simulation studies and real examples.

On the Fair Comparison of Optimization Algorithms in Different Machines

Sample size estimation for power and accuracy in the experimental comparison of algorithms

Best practices for comparing optimization algorithms

Two Possible Paradoxes In Numerical Comparisons Of Optimization Algorithms

Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances

On the Assessment of Benchmark Suites for Algorithm Comparison

Paradoxes in Numerical Comparison of Optimization Algorithms

An Experimental Comparison of Speed Scaling Algorithms with Deadline Feasibility Constraints

Computer Comparisons in the Presence of Performance Variation

The Limits of Assumption-free Tests for Algorithm Performance

Trial-Based Dominance Enables Non-Parametric Tests to Compare both the Speed and Accuracy of Stochastic Optimizers

Mean-based Borda Count for Paradox-free Comparisons of Optimization Algorithms

Benchmarking Derivative-Free Global Optimization Algorithms Under Limited Dimensions and Large Evaluation Budgets

A Comparison of Performance Measures for Online Algorithms

Optimization Algorithm Behavior Modeling: A Study On The Traveling Salesman Problem

A Novel Ranking Scheme for the Performance Analysis of Stochastic Optimization Algorithms using the Principles of Severity

Testing the Fairness-Accuracy Improvability of Algorithms

Paradox-Free Analysis for Comparing the Performance of Optimization Algorithms

Design and Analysis of Optimization Algorithms Using Computational Statistics

Two Criteria for Performance Analysis of Optimization Algorithms

A Statistical Comparison of Metaheuristics for Unrelated Parallel Machine Scheduling Problems with Setup Times