Abstract:GPUs are used for training, inference, and tuning the machine learning models. However, Deep Neural Network (DNN) vary widely in their ability to exploit the full power of high-performance GPUs. Spatial sharing of GPU enables multiplexing several DNNs on the GPU and can improve GPU utilization, thus improving throughput and lowering latency. DNN models given just the right amount of GPU resources can still provide low inference latency, just as much as dedicating all of the GPU for their inference task. An approach to improve DNN inference is tuning of the DNN model. Autotuning frameworks find the optimal low-level implementation for a certain target device based on the trained machine learning model, thus reducing the DNN's inference latency and increasing inference throughput. We observe an interdependency between the tuned model and its inference latency. A DNN model tuned with specific GPU resources provides the best inference latency when inferred with close to the same amount of GPU resources. While a model tuned with the maximum amount of the GPU's resources has poorer inference latency once the GPU resources are limited for inference. On the other hand, a model tuned with an appropriate amount of GPU resources still achieves good inference latency across a wide range of GPU resource availability. We explore the causes that impact the tuning of a model at different amounts of GPU resources. We present many techniques to maximize resource utilization and improve tuning performance. We enable controlled spatial sharing of GPU to multiplex several tuning applications on the GPU. We scale the tuning server instances and shard the tuning model across multiple client instances for concurrent tuning of different operators of a model, achieving better GPU multiplexing. With our improvements, we decrease DNN autotuning time by up to 75 percent and increase throughput by a factor of 5.

Bringing Auto-tuning to HIP: Analysis of Tuning Impact and Difficulty on AMD and Nvidia GPUs

Towards a Benchmarking Suite for Kernel Tuners

Performance Tuning for GPU-Embedded Systems: Machine-Learning-based and Analytical Model-driven Tuning Methodologies

Going green: optimizing GPUs for energy efficiency through model-steered auto-tuning

Preparing Ginkgo for AMD GPUs -- A Testimonial on Porting CUDA Code to HIP

HAOTuner: A Hardware Adaptive Operator Auto-Tuner for Dynamic Shape Tensor Compilers

Machine Learning Based Auto-tuning for Enhanced OpenCL Performance Portability

Searching CUDA code autotuning spaces with hardware performance counters: data from benchmarks running on various GPU architectures

Spatial Sharing of GPU for Autotuning DNN models

Analyzing Search Techniques for Autotuning Image-based GPU Kernels: The Impact of Sample Sizes

CLTune: A Generic Auto-Tuner for OpenCL Kernels

Machine Learning-driven Autotuning of Graphics Processing Unit Accelerated Computational Fluid Dynamics for Enhanced Performance

ytopt: Autotuning Scientific Applications for Energy Efficiency at Large Scales

Kernel Launcher: C++ Library for Optimal-Performance Portable CUDA Applications

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Stencil Computations on AMD and Nvidia Graphics Processors: Performance and Tuning Strategies

An Autotuning Protocol to Rapidly Build Autotuners

A Metaprogramming and Autotuning Framework for Deploying Deep Learning Applications

Performance Optimization using Multimodal Modeling and Heterogeneous GNN

A tool for top-down performance analysis of GPU-accelerated applications

Testing GPU Numerics: Finding Numerical Differences Between NVIDIA and AMD GPUs