Abstract:The GPU has emerged as the go-to accelerator for high throughput and parallel workloads, spanning scientific simulations to AI, thanks to its performance and power efficiency. Given that 6 out of the top 10 fastest supercomputers in the world use NVIDIA GPUs and many AI companies each employ 10,000's of NVIDIA GPUs, an accurate understanding of GPU power consumption is essential for making progress to further improve its efficiency. Despite the limited documentation and the lack of understanding of its mechanisms, NVIDIA GPUs' built-in power sensor, providing easily accessible power readings via the nvidia-smi interface, is widely used in energy efficient computing research on GPUs. Our study seeks to elucidate the internal mechanisms of the power readings provided by nvidia-smi and assess the accuracy of the power and energy consumption data. We have developed a suite of micro-benchmarks to profile the behaviour of nvidia-smi power readings and have evaluated them on over 70 different GPUs from all architectural generations since power measurement was first introduced in the 'Fermi' generation. We have identified several unforeseen problems in terms of power/energy measurement using nvidia-smi, for example on the A100 and H100 GPUs only 25% of the runtime is sampled for power consumption, during the other 75% of the time, the GPU can be using drastically different power and nvidia-smi and results presented by it are unaware of this. This along with other findings can lead to a drastic under/overestimation of energy consumed, especially when considering data centres housing tens of thousands of GPUs. We proposed several good practices that help to mitigate these problems. By comparing our results to those measured from an external power-meter, we have reduced the error in the energy measurement by an average of 35% and in some cases by as much as 65% in the test cases we present.

Programming-Level Power Measurement for GPU Clusters

Power-aware Programming with GPU Accelerators

Power Consumption Analysis of Parallel Algorithms on GPUs

Parallel Transient Stability-Constrained Optimal Power Flow Using GPU as Coprocessor.

GPGPU Memory Estimation and Optimization Targeting OpenCL Architecture

Part-time Power Measurements: nvidia-smi's Lack of Attention

Workload Analysis for Typical GPU Programs Using CUPTI Interface

AN PARALLEL AND DISTRIBUTED PROGRAMMING SOLUTION BASED ON HETEROGENEOUS GPU CLUSTER

Understanding the Impact of Input Entropy on FPU, CPU, and GPU Power

Input-Dependent Power Usage in GPUs

Power Profile Monitoring and Tracking Evolution of System-Wide HPC Workloads

Energy Efficient Real-Time Task Scheduling on CPU-GPU Hybrid Clusters

An Experimental GPU Global Memory Performance Estimation and Optimization

Optimizing Hardware Resource Partitioning and Job Allocations on Modern GPUs under Power Caps

Prediction of Performance and Power Consumption of GPGPU Applications

Characterizing the Execution Dynamics of GPGPU Applications

A Study of Performance Programming of CPU, GPU accelerated Computers and SIMD Architecture

Tools for GPU Computing - Debugging and Performance Analysis of Heterogenous HPC Applications

Performance Analysis of Parallel Programming Paradigms on CPU-GPU Clusters

Research and implementation of key technologies for grid computing based on GPU

High Performance Computing Via a GPU