Evaluating Portable Parallelization Strategies for Heterogeneous Architectures in High Energy Physics

Mohammad Atif,Meghna Battacharya,Paolo Calafiura,Taylor Childers,Mark Dewing,Zhihua Dong,Oliver Gutsche,Salman Habib,Kyle Knoepfel,Matti Kortelainen,Ka Hei Martin Kwok,Charles Leggett,Meifeng Lin,Vincent Pascuzzi,Alexei Strelchenko,Vakhtang Tsulaia,Brett Viren,Tianle Wang,Beomki Yeo,Haiwang Yu

2023-06-28

Abstract:High-energy physics (HEP) experiments have developed millions of lines of code over decades that are optimized to run on traditional x86 CPU systems. However, we are seeing a rapidly increasing fraction of floating point computing power in leadership-class computing facilities and traditional data centers coming from new accelerator architectures, such as GPUs. HEP experiments are now faced with the untenable prospect of rewriting millions of lines of x86 CPU code, for the increasingly dominant architectures found in these computational accelerators. This task is made more challenging by the architecture-specific languages and APIs promoted by manufacturers such as NVIDIA, Intel and AMD. Producing multiple, architecture-specific implementations is not a viable scenario, given the available person power and code maintenance issues. The Portable Parallelization Strategies team of the HEP Center for Computational Excellence is investigating the use of Kokkos, SYCL, OpenMP, std::execution::parallel and alpaka as potential portability solutions that promise to execute on multiple architectures from the same source code, using representative use cases from major HEP experiments, including the DUNE experiment of the Long Baseline Neutrino Facility, and the ATLAS and CMS experiments of the Large Hadron Collider. This cross-cutting evaluation of portability solutions using real applications will help inform and guide the HEP community when choosing their software and hardware suites for the next generation of experimental frameworks. We present the outcomes of our studies, including performance metrics, porting challenges, API evaluations, and build system integration.

High Energy Physics - Experiment,Distributed, Parallel, and Cluster Computing

What problem does this paper attempt to address?

The paper primarily discusses the challenges faced by high-energy physics (HEP) experiments when confronted with a shift in computational resource architectures and investigates various portable parallel strategies to address these challenges.

Specifically, as the proportion of floating-point computational power provided by new accelerator architectures (such as GPUs) in high-performance computing facilities and traditional data centers increases rapidly, HEP experiments are faced with the need to rewrite millions of lines of code originally optimized for x86 CPUs. This is not only a massive workload but also becomes impractical due to the different architecture-specific languages and APIs promoted by various manufacturers (such as NVIDIA, Intel, and AMD).

To tackle this issue, the portable parallel strategies team at the HEP Center of Computational Excellence evaluated several potential portable solutions, including Kokkos, SYCL, OpenMP, std::execution::parallel, and alpaka, which allow execution from the same source code on multiple different hardware architectures. Through representative case studies from large HEP experiments (including DUNE, ATLAS, and CMS experiments), these evaluations aim to help guide the HEP community in decision-making for future experimental framework software and hardware choices.

The paper details the selected portable layers, representative test cases, evaluation metrics considered, and performance assessment results of different portable layers. By conducting cross-domain evaluations of these portable layers, guidance can be provided to the HEP community regarding the selection of future software and hardware suites.

In summary, this paper attempts to address how to efficiently migrate the existing HEP experimental codebase to new heterogeneous computing architectures to meet the needs of future experiments while maintaining the maintainability and efficiency of the code.