Abstract:Performance variability, stemming from nondeterministic hardware and software behaviors or deterministic behaviors such as measurement bias, is a well-known phenomenon of computer systems which increases the difficulty of comparing computer performance metrics and is slated to become even more of a concern as interest in Big Data analytic increases. Conventional methods use various measures (such as geometric mean) to quantify the performance of different benchmarks to compare computers without considering this variability which may lead to wrong conclusions. In this paper, we propose three resampling methods for performance evaluation and comparison: a randomization test for a general performance comparison between two computers, bootstrapping confidence estimation, and an empirical distribution and five-number-summary for performance evaluation. The results show that for both PARSEC and high-variance BigDataBench benchmarks 1) the randomization test substantially improves our chance to identify the difference between performance comparisons when the difference is not large; 2) bootstrapping confidence estimation provides an accurate confidence interval for the performance comparison measure (e.g., ratio of geometric means); and 3) when the difference is very small, a single test is often not enough to reveal the nature of the computer performance due to the variability of computer systems.We further propose using empirical distribution to evaluate computer performance and a five-number-summary to summarize computer performance. We use published SPEC 2006 results to investigate the sources of performance variation by predicting performance and relative variation for 8,236 machines. We achieve a correlation of predicted performances of 0.992 and a correlation of predicted and measured relative variation of 0.5. Finally, we propose the utilization of a novel biplotting technique to visualize the effectiveness of benchmarks and cluster machines by behavior. We illustrate the results and conclusion through detailed Monte Carlo simulation studies and real examples.

Is Correlation Ranking Really Reliable for the Performance Counter Selection Conducted for Power Estimation?

A Cross-Layer Framework for Temporal Power and Supply Noise Prediction

Worst-case Power Integrity Prediction Using Convolutional Neural Network.

CORRELATION ANALYSIS OF PERFORMANCE METRICS FOR CLASSIFIER

Challenges in Variable Importance Ranking Under Correlation

A NEW CRITERION OF DETERMINING MEASUREMENT WEIGHTS IN POWER SYSTEM STATE ESTIMATION

Redundant Correlation Effect on Personalized Recommendation

Detecting Performance Anomaly with Correlation Analysis for Internetware

Modeling Link Correlation In Low-Power Wireless Networks

Performance Prediction under Power Capping

Investigating the deviation between prediction accuracy metrics and control performance metrics in the context of an ice-based thermal energy storage system

A Strategy on Selecting Performance Metrics for Classifier Evaluation.

A Correlation Analysis Method for Power Systems Based on Random Matrix Theory

Ranking Method for Peak-Load Shifting Considering Different Types of Data

Evaluating Neural Network Methods for PMC-based CPU Power Prediction

Power measurement techniques on standard compute nodes: A quantitative comparison

Computer Comparisons in the Presence of Performance Variation

Complete System Power Estimation Using Processor Performance Events

Many-Core Processors Granularity Evaluation by Considering Performance, Yield, and Lifetime Reliability

An empirical comparison of algorithmic, instruction, and architectural power prediction models for high performance embedded DSP processors

A Feature Selection Method Using Conditional Correlation Dispersion and Redundancy Analysis