Abstract:This paper investigates the multi-GPU performance of a 3D buoyancy driven cavity solver using MPI and OpenACC directives on different platforms. The paper shows that decomposing the total problem in different dimensions affects the strong scaling performance significantly for the GPU. Without proper performance optimizations, it is shown that 1D domain decomposition scales poorly on multiple GPUs due to the noncontiguous memory access. The performance using whatever decompositions can be benefited from a series of performance optimizations in the paper. Since the buoyancy driven cavity code is latency-bounded on the clusters examined, a series of optimizations both agnostic and tailored to the platforms are designed to reduce the latency cost and improve memory throughput between hosts and devices efficiently. First, the parallel message packing/unpacking strategy developed for noncontiguous data movement between hosts and devices improves the overall performance by about a factor of 2. Second, transferring different data based on the stencil sizes for different variables further reduces the communication overhead. These two optimizations are general enough to be beneficial to stencil computations having ghost changes on all of the clusters tested. Third, GPUDirect is used to improve the communication on clusters which have the hardware and software support for direct communication between GPUs without staging CPU's memory. Finally, overlapping the communication and computations is shown to be not efficient on multi-GPUs if only using MPI or MPI+OpenACC. Although we believe our implementation has revealed enough overlap, the actual running does not utilize the overlap well due to a lack of asynchronous progression.

Evaluating performance portability of five shared-memory programming models using a high-order unstructured CFD solver

Taking GPU Programming Models to Task for Performance Portability

Evaluation of performance portability frameworks for the implementation of a particle-in-cell code

Multi-GPU Performance Optimization of a CFD Code using OpenACC on Different Platforms

A Study of Performance Portability in Plasma Physics Simulations

OpenMP Performance Analysis of CFD Application on Intel Multicore and Manycore Architectures

Optimization and acceleration of flow simulations for CFD on CPU/GPU architecture

Hybrid programming-model strategies for GPU offloading of electronic structure calculation kernels

Performance Portability of Sparse Block Diagonal Matrix Multiple Vector Multiplications on GPUs

Performance Evaluation of Hybrid Programming Patterns for Large CPU/GPU Heterogeneous Clusters.

An approach to performance portability through generic programming

Performance optimizations for scalable CFD applications on hybrid CPU+MIC heterogeneous computing system with millions of cores

Massive parallelization and performance enhancement of an immersed boundary method based unsteady flow solver

Performance Optimization of a CFD Application on Intel Multicore and Manycore Architectures

Implementing Performance Portability of High Performance Computing Programs in the New Golden Age of Chip Architecture

CPU/GPU Heterogeneous Parallel CFD Solver and Optimizations

Evaluation of Programming Models and Performance for Stencil Computation on Current GPU Architectures

Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes

Performance and Power Efficient Massive Parallel Computational Model for HPC Heterogeneous Exascale Systems

Application of performance portability solutions for GPUs and many-core CPUs to track reconstruction kernels

An Evaluation and Comparison of GPU Hardware and Solver Libraries for Accelerating the OPM Flow Reservoir Simulator