Abstract:Performance variability, stemming from nondeterministic hardware and software behaviors or deterministic behaviors such as measurement bias, is a well-known phenomenon of computer systems which increases the difficulty of comparing computer performance metrics and is slated to become even more of a concern as interest in Big Data analytic increases. Conventional methods use various measures (such as geometric mean) to quantify the performance of different benchmarks to compare computers without considering this variability which may lead to wrong conclusions. In this paper, we propose three resampling methods for performance evaluation and comparison: a randomization test for a general performance comparison between two computers, bootstrapping confidence estimation, and an empirical distribution and five-number-summary for performance evaluation. The results show that for both PARSEC and high-variance BigDataBench benchmarks 1) the randomization test substantially improves our chance to identify the difference between performance comparisons when the difference is not large; 2) bootstrapping confidence estimation provides an accurate confidence interval for the performance comparison measure (e.g., ratio of geometric means); and 3) when the difference is very small, a single test is often not enough to reveal the nature of the computer performance due to the variability of computer systems.We further propose using empirical distribution to evaluate computer performance and a five-number-summary to summarize computer performance. We use published SPEC 2006 results to investigate the sources of performance variation by predicting performance and relative variation for 8,236 machines. We achieve a correlation of predicted performances of 0.992 and a correlation of predicted and measured relative variation of 0.5. Finally, we propose the utilization of a novel biplotting technique to visualize the effectiveness of benchmarks and cluster machines by behavior. We illustrate the results and conclusion through detailed Monte Carlo simulation studies and real examples.

2002 Performance / Price Sort and PennySort

Sort Race

P F ] 1 3 A ug 2 01 9 HPC AI 500 : A Benchmark Suite for HPC AI Systems

Architectual Implications of the NAS MG and FT Parallel Benchmarks

Data-centric workloads with MPI_Sort

Performance Evaluation of Parallel Sortings on the Supercomputer Fugaku

Characterizing and Optimizing TPC-C Workloads on Large-Scale Systems Using SSD Arrays

Computer Comparisons in the Presence of Performance Variation

PSM: A New Re-Ranking Algorithm for Named-Page

Exoshuffle-CloudSort

Parallel Sorting by Approximate Splitting for Multi-core Processors

High Performance Parallel Sort for Shared and Distributed Memory MIMD

Comparative of Advanced Sorting Algorithms (Quick Sort, Heap Sort, Merge Sort, Intro Sort, Radix Sort) Based on Time and Memory Usage

The Perfect Club Benchmarks: Effective Performance Evaluation of Supercomputers

Integer sorting on multicores: some (experiments and) observations

A study of integer sorting on multicores

A Creativity Survey of Parallel Sorting Algorithm

High-Performance and Flexible Parallel Algorithms for Semisort and Related Problems

Sorting with Asymmetric Read and Write Costs

Sorting it out in Hardware: A State-of-the-Art Survey