Abstract:Multiphase compressible flows are often characterized by a broad range of space and time scales, entailing large grids and small time steps. Simulations of these flows on CPU-based clusters can thus take several wall-clock days. Offloading the compute kernels to GPUs appears attractive but is memory-bound for many finite-volume and -difference methods, damping speedups. Even when realized, GPU-based kernels lead to more intrusive communication and I/O times owing to lower computation costs. We present a strategy for GPU acceleration of multiphase compressible flow solvers that addresses these challenges and obtains large speedups at scale. We use OpenACC for directive-based offloading of all compute kernels while maintaining low-level control when needed. An established Fortran preprocessor and metaprogramming tool, Fypp, enables otherwise hidden compile-time optimizations. This strategy exposes compile-time optimizations and high memory reuse while retaining readable, maintainable, and compact code. Remote direct memory access realized via CUDA-aware MPI and GPUDirect reduces halo-exchange communication time. We implement this approach in the open-source solver MFC [1] . Metaprogramming results in an 8-times speedup of the most expensive kernels compared to a statically compiled program, reaching 46% of peak FLOPs on modern NVIDIA GPUs and high arithmetic intensity (about 10 FLOPs/byte). In representative simulations, a single NVIDIA A100 GPU is 7-times faster compared to an Intel Xeon Cascade Lake (6248) CPU die, or about 300-times faster compared to a single such CPU core. At the same time, near-ideal (97%) weak scaling is observed for at least 13824 GPUs on OLCF Summit. A strong scaling efficiency of 84% is retained for an 8-times increase in GPU count. Collective I/O, implemented via MPI3, helps ensure the negligible contribution of data transfers ( <1% of the wall time for a typical, large simulation). Large many-GPU simulations of compressible (solid-)liquid-gas flows demonstrate the practical utility of this strategy.

Treecode and fast multipole method for N-body simulation with CUDA

Parallelized Implementation of the Finite Particle Method for Explicit Dynamics in GPU

Barnes-hut Treecode on GPU

FPGA Acceleration of TreePM N-body Simulations for Modified Newton Dynamics.

Accelerating the Finite Element Method for Calculating the Full 2-Body Problem with CUDA

High Performance Direct Gravitational N-body Simulations on Graphics Processing Units -- II: An implementation in CUDA

Acceleration for CFD Applications on Large GPU Clusters: an NPB Case Study

High Performance Matrix Multiplication on General Purpose Graphics Processing Units

Implementation of a Parallel Tree Method on a GPU

Data-Driven Execution of Fast Multipole Methods

Generalized Gpu Acceleration For Applications Employing Finite-Volume Methods

GPU-Based Implementation of Finite Element Method for Elasticity Using CUDA

Efficient molecular dynamics simulations with many-body potentials on graphics processing units

Accelerating Pythonic coupled cluster implementations: a comparison between CPUs and GPUs

Multidisciplinary simulation acceleration using multiple shared memory graphical processing units

Multi-GPU kinetic solvers using MPI and CUDA

Accelerating Fortran Codes: A Method for Integrating Coarray Fortran with CUDA Fortran and OpenMP

Parallel Time-Space Processing Model Based Fast N -Body Simulation on GPUs

Method for scalable and performant GPU-accelerated simulation of multiphase compressible flow

Streaming FFT Asynchronously on Graphics Processor Units

Optimizing Finite Volume Method Solvers on Nvidia GPUs.