Abstract:The GPU has emerged as the go-to accelerator for high throughput and parallel workloads, spanning scientific simulations to AI, thanks to its performance and power efficiency. Given that 6 out of the top 10 fastest supercomputers in the world use NVIDIA GPUs and many AI companies each employ 10,000's of NVIDIA GPUs, an accurate understanding of GPU power consumption is essential for making progress to further improve its efficiency. Despite the limited documentation and the lack of understanding of its mechanisms, NVIDIA GPUs' built-in power sensor, providing easily accessible power readings via the nvidia-smi interface, is widely used in energy efficient computing research on GPUs. Our study seeks to elucidate the internal mechanisms of the power readings provided by nvidia-smi and assess the accuracy of the power and energy consumption data. We have developed a suite of micro-benchmarks to profile the behaviour of nvidia-smi power readings and have evaluated them on over 70 different GPUs from all architectural generations since power measurement was first introduced in the 'Fermi' generation. We have identified several unforeseen problems in terms of power/energy measurement using nvidia-smi, for example on the A100 and H100 GPUs only 25% of the runtime is sampled for power consumption, during the other 75% of the time, the GPU can be using drastically different power and nvidia-smi and results presented by it are unaware of this. This along with other findings can lead to a drastic under/overestimation of energy consumed, especially when considering data centres housing tens of thousands of GPUs. We proposed several good practices that help to mitigate these problems. By comparing our results to those measured from an external power-meter, we have reduced the error in the energy measurement by an average of 35% and in some cases by as much as 65% in the test cases we present.

gpu_tracker: Python package for tracking and profiling GPU utilization in both desktop and high-performance computing environments

A tool for top-down performance analysis of GPU-accelerated applications

The Implementation of the Three-Dimensional Unified Gas-Kinetic Wave-Particle Method on Multiple Graphics Processing Units

Tools for top-down performance analysis of GPU-accelerated applications

Data-Driven Analysis to Understand GPU Hardware Resource Usage of Optimizations

Extending the ARC Information Providers to report information on GPU resources

ALICE HLT TPC Tracking of Pb-Pb Events on GPUs

Tools for GPU Computing - Debugging and Performance Analysis of Heterogenous HPC Applications

Parallel Transient Stability-Constrained Optimal Power Flow Using GPU as Coprocessor.

405 Acceleration of MARS by Using GPU

Real-Time Surface Topography Measurement with Graphic Processing Unit

Accelerating Genome-Wide Association Studies Using CUDA Compatible Graphics Processing Units

Workload Analysis for Typical GPU Programs Using CUPTI Interface

Exploring the Diversity of Multiple Job Deployments over GPUs for Efficient Resource Sharing

DeepProf: Performance Analysis for Deep Learning Applications via Mining GPU Execution Patterns

Parallel Data Mining on Graphics Processors

Efficient GPU Spatial-Temporal Multitasking

Study of A Gpu-Based Parallel Computing Method for the Monte Carlo Program

Part-time Power Measurements: nvidia-smi's Lack of Attention

Daisen: A Framework for Visualizing Detailed GPU Execution

Programming-Level Power Measurement for GPU Clusters