Abstract:As computing system become more complex, it is becoming harder for programmers to keep their codes optimized as the hardware gets updated. Autotuners try to alleviate this by hiding as many architecture-based optimization details as possible from the user, so that the code can be used efficiently across different generations of systems. In this article we introduce a new benchmark suite for evaluating the performance of optimization algorithms used by modern autotuners targeting GPUs. The suite contains tunable GPU kernels that are representative of real-world applications, allowing for comparisons between optimization algorithms and the examination of code optimization, search space difficulty, and performance portability. Our framework facilitates easy integration of new autotuners and benchmarks by defining a shared problem interface. Our benchmark suite is evaluated based on five characteristics: convergence rate, local minima centrality, optimal speedup, Permutation Feature Importance (PFI), and performance portability. The results show that optimization parameters greatly impact performance and the need for global optimization. The importance of each parameter is consistent across GPU architectures, however, the specific values need to be optimized for each architecture. Our portability study highlights the crucial importance of autotuning each application for a specific target architecture. The results reveal that simply transferring the optimal configuration from one architecture to another can result in a performance ranging from 58.5% to 99.9% of the optimal performance, depending on the GPU architecture. This highlights the importance of autotuning in modern computing systems and the value of our benchmark suite in facilitating the study of optimization algorithms and their effectiveness in achieving optimal performance for specific target architectures.

Gem5Tune: A Parameter Auto-tuning Framework for Gem5 Simulator to Reduce Errors

Performance Evaluation of Algorithms for Optimizing Processor Simulator Parameters

Performance Error Evaluation of gem5 Simulator for ARM Server.

BestConfig: Tapping the Performance Potential of Systems Via Automatic Configuration Tuning

Parallel computing based parameter auto-tuning algorithm for optimization solvers

ML$^2$Tuner: Efficient Code Tuning via Multi-Level Machine Learning Models

Compiler Auto-tuning through Multiple Phase Learning

Compiler Autotuning through Multiple Phase Learning

Multi-Intention-Aware Configuration Selection for Performance Tuning

ytopt: Autotuning Scientific Applications for Energy Efficiency at Large Scales

parti-gem5: gem5's Timing Mode Parallelised

MindOpt Tuner: Boost the Performance of Numerical Software by Automatic Parameter Tuning

An Autotuning Protocol to Rapidly Build Autotuners

A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning

Taming the "Monster": Overcoming Program Optimization Challenges on SW26010 Through Precise Performance Modeling

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Towards a Benchmarking Suite for Kernel Tuners

Efficient Parameter Tuning for a Structure-Based Virtual Screening HPC Application

Deep Parameter Optimisation

Techniques for CAD Tool Parameter Auto-tuning in Physical Synthesis - A Survey (invited Paper).