Abstract:This paper investigates the multi-GPU performance of a 3D buoyancy driven cavity solver using MPI and OpenACC directives on different platforms. The paper shows that decomposing the total problem in different dimensions affects the strong scaling performance significantly for the GPU. Without proper performance optimizations, it is shown that 1D domain decomposition scales poorly on multiple GPUs due to the noncontiguous memory access. The performance using whatever decompositions can be benefited from a series of performance optimizations in the paper. Since the buoyancy driven cavity code is latency-bounded on the clusters examined, a series of optimizations both agnostic and tailored to the platforms are designed to reduce the latency cost and improve memory throughput between hosts and devices efficiently. First, the parallel message packing/unpacking strategy developed for noncontiguous data movement between hosts and devices improves the overall performance by about a factor of 2. Second, transferring different data based on the stencil sizes for different variables further reduces the communication overhead. These two optimizations are general enough to be beneficial to stencil computations having ghost changes on all of the clusters tested. Third, GPUDirect is used to improve the communication on clusters which have the hardware and software support for direct communication between GPUs without staging CPU's memory. Finally, overlapping the communication and computations is shown to be not efficient on multi-GPUs if only using MPI or MPI+OpenACC. Although we believe our implementation has revealed enough overlap, the actual running does not utilize the overlap well due to a lack of asynchronous progression.

Performance Modeling and Optimization of Parallel LU-SGS on Many-Core Processors for 3D High-Order CFD Simulations

An Efficient Wavefront Parallel Algorithm for Structured Three Dimensional Lu-Sgs

A Parallel Function Evaluation Approach for Solution to Large-Scale Equation-Oriented Models.

Characterize and Optimize Dense Linear Solver on Multi-core CPUs

A Hierarchical Grid Algorithm for Accelerating High-Performance Conjugate Gradient Benchmark on Sunway Many-Core Processor

Parallel Implementation of the Four-Dimensional Lattice Spring Model on Heterogeneous CPU-GPU Systems

Performance optimizations for scalable CFD applications on hybrid CPU+MIC heterogeneous computing system with millions of cores

Efficient Parallel Implementation of the Lattice Boltzmann Method on Large Clusters of Graphic Processing Units

NUMERICAL SIMULATION OF PLANETARY FLUID DYNAMICS ON CPU-MIC HETEROGENEOUS MANY-CORE SYSTEMS

Acceleration of a Production-Level Unstructured Grid Finite Volume CFD Code on GPU

Parallel Sparse LU Factorization With Machine-Learning Method on Multi-core Processors

Evaluating performance portability of five shared-memory programming models using a high-order unstructured CFD solver

Parallel Performance Modeling of Irregular Applications in Cell-Centered Finite Volume Methods over Unstructured Tetrahedral Meshes

Optimizing Multi-grid Computation and Parallelization on Multi-cores.

OpenMP Performance Analysis of CFD Application on Intel Multicore and Manycore Architectures

Optimization and acceleration of flow simulations for CFD on CPU/GPU architecture

Utilizing Multiple Xeon Phi Coprocessors on One Compute Node.

Parallel MCGLS and ICGLS Methods for Least Squares Problems on Distributed Memory Architectures

Multi-GPU Performance Optimization of a CFD Code using OpenACC on Different Platforms

Optimization Strategies for Multi‐block Structured CFD Simulation Based on Sunway TaihuLight

Improving Performance for Simulating Complex Fluids on Massively Parallel Computers by Component Loop-Unrolling and Communication Hiding