Abstract:This work presents GALÆXI as a novel, energy-efficient flow solver for the simulation of compressible flows on unstructured hexahedral meshes leveraging the parallel computing power of modern Graphics Processing Units (GPUs). GALÆXI implements the high-order Discontinuous Galerkin Spectral Element Method (DGSEM) using shock capturing with a finite-volume subcell approach to ensure the stability of the high-order scheme near shocks. This work provides details on the general code design, the parallelization strategy, and the implementation approach for the compute kernels with a focus on the element local mappings between volume and surface data due to the unstructured mesh. The scheme is implemented using a pure distributed memory parallelization based on a domain decomposition, where each GPU handles a distinct region of the computational domain. On each GPU, the computations are assigned to different compute streams which allows to antedate the computation of quantities required for communication while performing local computations from other streams to hide the communication latency. This parallelization strategy allows for maximizing the use of available computational resources. This results in excellent strong scaling properties of GALÆXI up to 1024 GPUs if each GPU is assigned a minimum of one million degrees of freedom. To verify its implementation, a convergence study is performed that recovers the theoretical order of convergence of the implemented numerical schemes. Moreover, the solver is validated using both the incompressible and compressible formulation of the Taylor–Green-Vortex at a Mach number of 0.1 and 1.25, respectively. A mesh convergence study shows that the results converge to the high-fidelity reference solution and that the results match the original CPU implementation. Finally, GALÆXI is applied to a large-scale wall-resolved large eddy simulation of a linear cascade of the NASA Rotor 37. Here, the supersonic region and shocks at the leading edge are captured accurately and robustly by the implemented shock-capturing approach. It is demonstrated that GALÆXI requires less than half of the energy to carry out this simulation in comparison to the reference CPU implementation. This renders GALÆXI as a potent tool for accurate and efficient simulations of compressible flows in the realm of exascale computing and the associated new HPC architectures.

IDEFIX: a versatile performance-portable Godunov code for astrophysical flows

Heterogeneous Programming and Optimization of Gyrokinetic Toroidal Code and Large-Scale Performance Test on TH-1A.

GALÆXI: Solving complex compressible flows with high-order discontinuous Galerkin methods on accelerator-based systems

A GPU-Accelerated Modern Fortran Version of the ECHO Code for Relativistic Magnetohydrodynamics

A hybrid CPU‐GPU paradigm to accelerate reactive CFD simulations

Method for scalable and performant GPU-accelerated simulation of multiphase compressible flow

Method for portable, scalable, and performant GPU-accelerated simulation of multiphase compressible flow

OpenACC acceleration of an unstructured CFD solver based on a reconstructed discontinuous Galerkin method for compressible flows

Matrix-free approaches for GPU acceleration of a high-order finite element hydrodynamics application using MFEM, Umpire, and RAJA

A GPU-ready pseudo-spectral method for direct numerical simulations of multiphase turbulence

An adaptive finite element multigrid solver using GPU acceleration

Accuracy and scalability of incompressible inductionless MHD codes applied to fusion technologies

GPU-accelerated 3D neutron diffusion code based on finite difference method

Modern Gyrokinetic Particle-In-Cell Simulation of Fusion Plasmas on Top Supercomputers

DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization

FISH: A 3D parallel MHD code for astrophysical applications

JAX-Fluids 2.0: Towards HPC for Differentiable CFD of Compressible Two-phase Flows

JefiGPU: Jefimenko's equations on GPU

Efficient implementation of ADER discontinuous Galerkin schemes for a scalable hyperbolic PDE engine

Robust and effective ab initio molecular dynamics simulations on the GPU cloud infrastructure using the Schrödinger Materials Science Suite