Abstract:Multiphase compressible flows are often characterized by a broad range of space and time scales, entailing large grids and small time steps. Simulations of these flows on CPU-based clusters can thus take several wall-clock days. Offloading the compute kernels to GPUs appears attractive but is memory-bound for many finite-volume and -difference methods, damping speedups. Even when realized, GPU-based kernels lead to more intrusive communication and I/O times owing to lower computation costs. We present a strategy for GPU acceleration of multiphase compressible flow solvers that addresses these challenges and obtains large speedups at scale. We use OpenACC for directive-based offloading of all compute kernels while maintaining low-level control when needed. An established Fortran preprocessor and metaprogramming tool, Fypp, enables otherwise hidden compile-time optimizations. This strategy exposes compile-time optimizations and high memory reuse while retaining readable, maintainable, and compact code. Remote direct memory access realized via CUDA-aware MPI and GPUDirect reduces halo-exchange communication time. We implement this approach in the open-source solver MFC [1] . Metaprogramming results in an 8-times speedup of the most expensive kernels compared to a statically compiled program, reaching 46% of peak FLOPs on modern NVIDIA GPUs and high arithmetic intensity (about 10 FLOPs/byte). In representative simulations, a single NVIDIA A100 GPU is 7-times faster compared to an Intel Xeon Cascade Lake (6248) CPU die, or about 300-times faster compared to a single such CPU core. At the same time, near-ideal (97%) weak scaling is observed for at least 13824 GPUs on OLCF Summit. A strong scaling efficiency of 84% is retained for an 8-times increase in GPU count. Collective I/O, implemented via MPI3, helps ensure the negligible contribution of data transfers ( <1% of the wall time for a typical, large simulation). Large many-GPU simulations of compressible (solid-)liquid-gas flows demonstrate the practical utility of this strategy.

Improving Performance for Simulating Complex Fluids on Massively Parallel Computers by Component Loop-Unrolling and Communication Hiding

A Parallel Simulator for Massive Reservoir Models Utilizing Distributed-Memory Parallel Systems

Parallelized Implementation of the Finite Particle Method for Explicit Dynamics in GPU

Parallel Algorithms for Moving Boundary Problems by Local Remeshing

An efficient communication strategy for massively parallel computation in CFD

Performance optimizations for scalable CFD applications on hybrid CPU+MIC heterogeneous computing system with millions of cores

Optimization and acceleration of flow simulations for CFD on CPU/GPU architecture

swHPFM: Refactoring and Optimizing the Structured Grid Fluid Mechanical Algorithm on the Sunway TaihuLight Supercomputer

A PARALLEL SOLVING DESIGN FOR UNSTEADY NAVIER-STOKES EQUATION

GPU Optimization for High-Quality Kinetic Fluid Simulation

Performance Optimization of Planetary Fluid Dynamics Numerical Simulation Program on Thousands Cores

A parallel solver framework for fully implicit monolithic fluid-structure interaction

Performance modeling and running strategy of parallel cdugksFOAM program

An efficient parallel algorithm with application to computational fluid dynamics

Parallel Scheme for Multi-Layer Refinement Non-Uniform Grid Lattice Boltzmann Method Based on Load Balancing

Achieving high performance and portable parallel GMRES algorithm for compressible flow simulations on unstructured grids

Optimization Strategies for Multi‐block Structured CFD Simulation Based on Sunway TaihuLight

Accelerating CFD simulation with high order finite difference method on curvilinear coordinates for modern GPU clusters

Massive parallelization and performance enhancement of an immersed boundary method based unsteady flow solver

Efficient Same-Dimensional Implicit Time Advancement Parallel Scheme and Optimization Methods for the Iteration Parameters Using a Graphics-Processing Unit

Method for scalable and performant GPU-accelerated simulation of multiphase compressible flow