CaLES: A GPU-accelerated solver for large-eddy simulation of wall-bounded flows

Maochao Xiao,Alessandro Ceci,Pedro Costa,Johan Larsson,Sergio Pirozzoli

2024-11-15

Abstract:We introduce CaLES, a GPU-accelerated finite-difference solver designed for large-eddy simulations (LES) of incompressible wall-bounded flows in massively parallel environments. Built upon the existing direct numerical simulation (DNS) solver CaNS, CaLES relies on low-storage, third-order Runge-Kutta schemes for temporal discretization, with the option to treat viscous terms via an implicit Crank-Nicolson scheme in one or three directions. A fast direct solver, based on eigenfunction expansions, is used to solve the discretized Poisson/Helmholtz equations. For turbulence modeling, the classical Smagorinsky model with van Driest near-wall damping and the dynamic Smagorinsky model are implemented, along with a logarithmic law wall model. GPU acceleration is achieved through OpenACC directives, following CaNS-2.3.0. Performance assessments were conducted on the Leonardo cluster at CINECA, Italy. Each node is equipped with one Intel Xeon Platinum 8358 CPU (2.60 GHz, 32 cores) and four NVIDIA A100 GPUs (64 GB HBM2e), interconnected via NVLink 3.0 (200 GB/s). The inter-node communication bandwidth is 25 GB/s, supported by a DragonFly+ network architecture with NVIDIA Mellanox InfiniBand HDR. Results indicate that the computational speed on a single GPU is equivalent to approximately 15 CPU nodes, depending on the treatment of viscous terms and the subgrid-scale model, and that the solver efficiently scales across multiple GPUs. The predictive capability of CaLES has been tested using multiple flow cases, including decaying isotropic turbulence, turbulent channel flow, and turbulent duct flow. The high computational efficiency of the solver enables grid convergence studies on extremely fine grids, pinpointing non-monotonic grid convergence for wall-modeled LES.

Fluid Dynamics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to develop an efficient large - eddy simulation (LES) solver suitable for wall - bounded fluids in a massively parallel environment, in order to balance the computational cost and prediction accuracy. Specifically, the paper introduces a GPU - accelerated finite - difference solver named CaLES, which is built based on the existing direct numerical simulation (DNS) solver CaNS, aiming to improve the computational efficiency of LES through the following improvements:

1. **Time discretization**: A low - storage third - order Runge - Kutta scheme is used for time discretization, and an option is provided to use the implicit Crank - Nicolson scheme to handle the viscous terms to improve stability and accuracy.
2. **Fast solver**: A fast direct solver based on eigenfunction expansion is used to solve the discretized Poisson/Helmholtz equation, thus speeding up the calculation.
3. **Turbulence model**: The classical Smagorinsky model (with van Driest near - wall damping), the dynamic Smagorinsky model, and the logarithmic - law wall model are implemented to better simulate turbulent phenomena.
4. **GPU acceleration**: GPU acceleration is achieved through OpenACC instructions, significantly improving the computational efficiency.

The paper also evaluates the performance of CaLES on a high - performance computing cluster. The results show that the computing speed of a single GPU is equivalent to the computing power of about 15 CPU nodes, and the solver has good scalability on multiple GPUs. In addition, the prediction ability of CaLES has been tested through various flow cases, including decaying isotropic turbulence, channel turbulence, and pipe turbulence, verifying its non - monotonic grid convergence characteristics on high - resolution grids.

In summary, the main objective of this paper is to develop an efficient, scalable GPU - accelerated LES solver to support high - precision simulations of complex wall - bounded fluids, especially for applications at high Reynolds numbers.