Abstract:Multiphase compressible flows are often characterized by a broad range of space and time scales, entailing large grids and small time steps. Simulations of these flows on CPU-based clusters can thus take several wall-clock days. Offloading the compute kernels to GPUs appears attractive but is memory-bound for many finite-volume and -difference methods, damping speedups. Even when realized, GPU-based kernels lead to more intrusive communication and I/O times owing to lower computation costs. We present a strategy for GPU acceleration of multiphase compressible flow solvers that addresses these challenges and obtains large speedups at scale. We use OpenACC for directive-based offloading of all compute kernels while maintaining low-level control when needed. An established Fortran preprocessor and metaprogramming tool, Fypp, enables otherwise hidden compile-time optimizations. This strategy exposes compile-time optimizations and high memory reuse while retaining readable, maintainable, and compact code. Remote direct memory access realized via CUDA-aware MPI and GPUDirect reduces halo-exchange communication time. We implement this approach in the open-source solver MFC [1] . Metaprogramming results in an 8-times speedup of the most expensive kernels compared to a statically compiled program, reaching 46% of peak FLOPs on modern NVIDIA GPUs and high arithmetic intensity (about 10 FLOPs/byte). In representative simulations, a single NVIDIA A100 GPU is 7-times faster compared to an Intel Xeon Cascade Lake (6248) CPU die, or about 300-times faster compared to a single such CPU core. At the same time, near-ideal (97%) weak scaling is observed for at least 13824 GPUs on OLCF Summit. A strong scaling efficiency of 84% is retained for an 8-times increase in GPU count. Collective I/O, implemented via MPI3, helps ensure the negligible contribution of data transfers ( <1% of the wall time for a typical, large simulation). Large many-GPU simulations of compressible (solid-)liquid-gas flows demonstrate the practical utility of this strategy.

GPU acceleration of FSI simulations by the immersed boundary-lattice Boltzmann coupling scheme

Parallelized Implementation of the Finite Particle Method for Explicit Dynamics in GPU

The Immersed Boundary-Lattice Boltzmann Method Parallel Model for Fluid-Structure Interaction on Heterogeneous Platforms

Implementation of Multi-GPU Based Lattice Boltzmann Method for Flow Through Porous Media

An efficient flux-reconstructed lattice boltzmann flux solver for flow interaction of multi-structure with curved boundary

404 GPU Acceleration for 3D-LBM Incompressible-Flow Solver

Implementation of a direct-addressing based lattice Boltzmann GPU solver for multiphase flow in porous media

ICCM2016: the Implementation of Two-Dimensional Multi-Block Lattice Boltzmann Method on GPU

Accelerated Lattice Boltzmann Simulation Using GPU and OpenACC with Data Management

Accelerating Lattice-Boltzmann Method with Multi-GPUs

The Implementation of Multi-Block Lattice Boltzmann Method on GPU

Implementation and Optimization of Lattice Boltzmann Method for Fluid Flow on GPU with CUDA

Gpu Speed-Up For The Implicit Navier-Stokes Solver

Optimization and acceleration of flow simulations for CFD on CPU/GPU architecture

Performance engineering for the Lattice Boltzmann method on GPGPUs: Architectural requirements and performance results

Method for scalable and performant GPU-accelerated simulation of multiphase compressible flow

Massive parallelization and performance enhancement of an immersed boundary method based unsteady flow solver

A Simple One-Step Index Algorithm for Implementation of Lattice Boltzmann Method on GPU

A Relaxed Multi-Direct-forcing Immersed Boundary-Cascaded Lattice Boltzmann Method Accelerated on GPU

GPU implementation of lattice Boltzmann method for flows with curved boundaries

Multi-GPU Based Lattice Boltzmann Method for Hemodynamic Simulation in Patient-Specific Cerebral Aneurysm