Abstract:Programming and optimizing complex, real-world CFD codes on current many-core accelerated HPC systems is very challenging, especially when collaborating CPUs and accelerators to fully tap the potential of heterogeneous systems. In this paper, with a tri-level hybrid and heterogeneous programming model using MPI + OpenMP + CUDA, we port and optimize our high-order multi-block structured CFD software HOSTA on the GPU-accelerated TianHe-1A supercomputer. HOSTA adopts two self-developed high-order compact definite difference schemes WCNS and HDCS that can simulate flows with complex geometries. We present a dual-level parallelization scheme for efficient multi-block computation on GPUs and perform particular kernel optimizations for high-order CFD schemes. The GPU-only approach achieves a speedup of about 1.3 when comparing one Tesla M2050 GPU with two Xeon X5670 CPUs. To achieve a greater speedup, we collaborate CPU and GPU for HOSTA instead of using a naive GPU-only approach. We present a novel scheme to balance the loads between the store-poor GPU and the store-rich CPU. Taking CPU and GPU load balance into account, we improve the maximum simulation problem size per TianHe-1A node for HOSTA by 2.3×, meanwhile the collaborative approach can improve the performance by around 45% compared to the GPU-only approach. Further, to scale HOSTA on TianHe-1A, we propose a gather/scatter optimization to minimize PCI-e data transfer times for ghost and singularity data of 3D grid blocks, and overlap the collaborative computation and communication as far as possible using some advanced CUDA and MPI features. Scalability tests show that HOSTA can achieve a parallel efficiency of above 60% on 1024 TianHe-1A nodes. With our method, we have successfully simulated an EET high-lift airfoil configuration containing 800M cells and China's large civil airplane configuration containing 150M cells. To our best knowledge, those are the largest-scale CPU–GPU collaborative simulations that solve realistic CFD problems with both complex configurations and high-order schemes.

Large-Scale Parallelization and Optimization of Lattice QCD on Tianhe New Generation Supercomputer

Analysis and MPI Implementation of LQCD Dslash on Sunway TaihuLight*

Large-Scale Parallelization of Lattice QCD on Sunway TaihuLight Supercomputer

Heterogeneous Programming and Optimization of Gyrokinetic Toroidal Code and Large-Scale Performance Test on TH-1A.

Accelerating Lattice QCD on Sunway Many-core Processor

Parallelizing and Optimizing Large‐scale 3D Multi‐phase Flow Simulations on the Tianhe‐2 Supercomputer

623 Tflop/s HPCG Run on Tianhe-2: Leveraging Millions of Hybrid Cores.

Large-Scale Heterogeneous Computing for 3D Deterministic Particle Transport on Tianhe-2A Supercomputer

Parallel algorithm design and optimization of geodynamic numerical simulation application on the Tianhe new-generation high-performance computer

Transplantation and optimization of molecular dynamics simulation on MT-3000

Heterogeneous Computing and Optimization on Tianhe-2 Supercomputer System for High-Order Accurate CFD Applications

Efficient Parallel Computing and Performance Tuning for Multi-Block Structured Grid CFD Applications on Tianhe Supercomputer

Optimization of A Parallel Cfd Code and Its Performance Evaluation on Tianhe-1a

Efficient Parallel Implementation of Large Scale 3D Structured Grid CFD Applications on the Tianhe-1A Supercomputer

Parallel Supercomputing PC Cluster and Some Physical Results in Lattice QCD

Multi-GPU Hybrid Programming Accelerated Three-Dimensional Phase-Field Model in Binary Alloy

Performance optimizations for scalable CFD applications on hybrid CPU+MIC heterogeneous computing system with millions of cores

Exploiting Hierarchy Parallelism for Molecular Dynamics on a Petascale Heterogeneous System

Collaborating CPU and GPU for Large-Scale High-Order CFD Simulations with Complex Grids on the TianHe-1A Supercomputer

Accelerating High-Order CFD Simulations for Multi-block Structured Grids on the TianHe-1A Supercomputer

High performance Beowulf computer for lattice QCD