Abstract:Performance variability, stemming from nondeterministic hardware and software behaviors or deterministic behaviors such as measurement bias, is a well-known phenomenon of computer systems which increases the difficulty of comparing computer performance metrics and is slated to become even more of a concern as interest in Big Data analytic increases. Conventional methods use various measures (such as geometric mean) to quantify the performance of different benchmarks to compare computers without considering this variability which may lead to wrong conclusions. In this paper, we propose three resampling methods for performance evaluation and comparison: a randomization test for a general performance comparison between two computers, bootstrapping confidence estimation, and an empirical distribution and five-number-summary for performance evaluation. The results show that for both PARSEC and high-variance BigDataBench benchmarks 1) the randomization test substantially improves our chance to identify the difference between performance comparisons when the difference is not large; 2) bootstrapping confidence estimation provides an accurate confidence interval for the performance comparison measure (e.g., ratio of geometric means); and 3) when the difference is very small, a single test is often not enough to reveal the nature of the computer performance due to the variability of computer systems.We further propose using empirical distribution to evaluate computer performance and a five-number-summary to summarize computer performance. We use published SPEC 2006 results to investigate the sources of performance variation by predicting performance and relative variation for 8,236 machines. We achieve a correlation of predicted performances of 0.992 and a correlation of predicted and measured relative variation of 0.5. Finally, we propose the utilization of a novel biplotting technique to visualize the effectiveness of benchmarks and cluster machines by behavior. We illustrate the results and conclusion through detailed Monte Carlo simulation studies and real examples.

Benchmarking historical corporate performance

Nonparametric Bayesian multiple testing for longitudinal performance stratification

Good, great, or lucky? Screening for firms with sustained superior performance using heavy-tailed priors

Fully Bayesian Benchmarking of Small Area Estimation Models

A Computationally Efficient Approach to Fully Bayesian Benchmarking

Who Are Your Peers? An Empirical Investigation Of The Matched Sample Comparison Analysis

Bayesian probabilistic models for corporate context, with an application to internal audit activities

Benchmarking performance indicators for banks

Computer Comparisons in the Presence of Performance Variation

Benchmarking M6 Competitors: An Analysis of Financial Metrics and Discussion of Incentives

Less Information, More Comparison, and Better Performance: Evidence from a Field Experiment

Cross comparison and modelling of Goldman Sachs, Morgan Stanley, JPMorgan Chase, Bank of America, and Franklin Resources

A Tree-Based Approach for Addressing Self-Selection in Impact Studies with Big Data

Bayesian Inference for Comparative Research

Evaluating the resource management and profitability efficiencies of US commercial banks from a dynamic network perspective

Use of active peer benchmarks in assessing UK mutual fund performance and performance persistence

The partition problem: case studies in Bayesian screening for time-varying model structure

Benchmarking alternative interpretable machine learning models for corporate probability of default

Benchmark-based deviation and drawdown measures in portfolio optimization

Does a Bayesian Approach Generate Robust Forecasts? Evidence from Applications in Portfolio Investment Decisions

A comparison of Bayesian, Hazard, and Mixed Logit model of bankruptcy prediction