Abstract:Current supercomputers often have a heterogeneous architecture using both CPUs and GPUs. At the same time, numerical simulation tasks frequently involve multiphysics scenarios whose components run on different hardware due to multiple reasons, e.g., architectural requirements, pragmatism, etc. This leads naturally to a software design where different simulation modules are mapped to different subsystems of the heterogeneous architecture. We present a detailed performance analysis for such a hybrid four-way coupled simulation of a fully resolved particle-laden flow. The Eulerian representation of the flow utilizes GPUs, while the Lagrangian model for the particles runs on CPUs. First, a roofline model is employed to predict the node level performance and to show that the lattice-Boltzmann-based fluid simulation reaches very good performance on a single GPU. Furthermore, the GPU-GPU communication for a large-scale flow simulation results in only moderate slowdowns due to the efficiency of the CUDA-aware MPI communication, combined with communication hiding techniques. On 1024 A100 GPUs, a parallel efficiency of up to 71% is achieved. While the flow simulation has good performance characteristics, the integration of the stiff Lagrangian particle system requires frequent CPU-CPU communications that can become a bottleneck. Additionally, special attention is paid to the CPU-GPU communication overhead since this is essential for coupling the particles to the flow simulation. However, thanks to our problem-aware co-partitioning, the CPU-GPU communication overhead is found to be negligible. As a lesson learned from this development, four criteria are postulated that a hybrid implementation must meet for the efficient use of heterogeneous supercomputers. Additionally, an a priori estimate of the speedup for hybrid implementations is suggested.

Scaling Up Multiphysics

Large Scale Numerical Simulation Via Parallelization and Reconfigurable Computing Hardware

Multiscale Fluid Mechanics and Modeling

A Parallel Simulator for Massive Reservoir Models Utilizing Distributed-Memory Parallel Systems

An Efficient Domain Decomposition Method in Multi-physics Simulation of Package Systems

Multiphysics Computing of Challenging Antenna Arrays under a Supercomputer Framework

Coupled Geomechanics and Reservoir Flow Modeling on Distributed Memory Parallel Computers

Multi-physics simulations using a hierarchical interchangeable software interface

Scalable and fast heterogeneous molecular simulation with predictive parallelization schemes

Parallel visualization of large-scale multifield scientific data

A parallel dual-grid multiscale approach to CFD-DEM couplings

MOOSE: Enabling Massively Parallel Multiphysics Simulation

An Efficient and Parallel Scalable Geomechanics Simulator for Reservoir Simulation

Multiphysics for nuclear energy applications using a cohesive computational framework

A scalable Euler–Lagrange approach for multiphase flow simulation on spectral elements

Scalable computational kernels for mortar finite element methods

Efficiency and scalability of fully-resolved fluid-particle simulations on heterogeneous CPU-GPU architectures

Parallelization of Kinetic Theory Simulations

Scalability of high-performance PDE solvers

Multiphase flows: Rich physics, challenging theory, and big simulations

Multi-scale Modeling and Computation