Abstract:Programming and optimizing complex, real-world CFD codes on current many-core accelerated HPC systems is very challenging, especially when collaborating CPUs and accelerators to fully tap the potential of heterogeneous systems. In this paper, with a tri-level hybrid and heterogeneous programming model using MPI + OpenMP + CUDA, we port and optimize our high-order multi-block structured CFD software HOSTA on the GPU-accelerated TianHe-1A supercomputer. HOSTA adopts two self-developed high-order compact definite difference schemes WCNS and HDCS that can simulate flows with complex geometries. We present a dual-level parallelization scheme for efficient multi-block computation on GPUs and perform particular kernel optimizations for high-order CFD schemes. The GPU-only approach achieves a speedup of about 1.3 when comparing one Tesla M2050 GPU with two Xeon X5670 CPUs. To achieve a greater speedup, we collaborate CPU and GPU for HOSTA instead of using a naive GPU-only approach. We present a novel scheme to balance the loads between the store-poor GPU and the store-rich CPU. Taking CPU and GPU load balance into account, we improve the maximum simulation problem size per TianHe-1A node for HOSTA by 2.3×, meanwhile the collaborative approach can improve the performance by around 45% compared to the GPU-only approach. Further, to scale HOSTA on TianHe-1A, we propose a gather/scatter optimization to minimize PCI-e data transfer times for ghost and singularity data of 3D grid blocks, and overlap the collaborative computation and communication as far as possible using some advanced CUDA and MPI features. Scalability tests show that HOSTA can achieve a parallel efficiency of above 60% on 1024 TianHe-1A nodes. With our method, we have successfully simulated an EET high-lift airfoil configuration containing 800M cells and China's large civil airplane configuration containing 150M cells. To our best knowledge, those are the largest-scale CPU–GPU collaborative simulations that solve realistic CFD problems with both complex configurations and high-order schemes.

CPU/GPU Heterogeneous Parallel CFD Solver and Optimizations

A CFD Heterogeneous Parallel Solver Based on Collaborating CPU and GPU

Optimization and acceleration of flow simulations for CFD on CPU/GPU architecture

Development of GPU-based Parallel CFD Solver for Unstructured Meshes

Hybrid MPI and CUDA Parallelization for CFD Applications on Multi-GPU HPC Clusters

Accelerating CFD simulation with high order finite difference method on curvilinear coordinates for modern GPU clusters

GPU-Accelerated Computations for Supersonic Flow Modeling on Hybrid Grids

A Cfd Solver On Graphical Processing Unites For Turbulence Simulations

A Multi-GPU Parallel Algorithm in Hypersonic Flow Computations

Massive parallelization and performance enhancement of an immersed boundary method based unsteady flow solver

GPU Acceleration of CFD Algorithm: HSMAC and SIMPLE.

Parallel computation of compressible turbulence using multi-GPU clusters

OpenACC acceleration of an unstructured CFD solver based on a reconstructed discontinuous Galerkin method for compressible flows

CPU-GPU Heterogeneous Code Acceleration of a Finite Volume Computational Fluid Dynamics Solver

Implementation and Optimization of Lattice Boltzmann Method for Fluid Flow on GPU with CUDA

Gpu Speed-Up For The Implicit Navier-Stokes Solver

A novel GPU-parallelized meshless method for solving compressible turbulent flows

Collaborating CPU and GPU for Large-Scale High-Order CFD Simulations with Complex Grids on the TianHe-1A Supercomputer

Development of A GPU Based Unstructured Solver for High Fidelity Turbomachinery Simulation

An Efficient Acceleration of Solving Heat and Mass Transfer Equations with the Second Kind Boundary Conditions in Capillary Porous Radially Composite Cylinder Using Programmable Graphics Hardware

The Numerical Solutions For Navier-Stokes Equations By Gpu