Abstract:Heterogeneous nodes composed of a multicore CPU and accelerators are today's norm in high-performance computing (HPC) platforms due to their superior performance and energy efficiency. Tools such as OpenCL and hybrid combinations such as OpenMP plus OpenACC are used for developing portable parallel programs for such nodes. However, these tools have some drawbacks, including a lack of compiler support for nested parallelism, performance portability, automatic heterogeneous workload distribution, user-friendly thread placement, and processor affinity essential to the portable performance of hybrid programs executing on such nodes. In this paper, we propose OpenH, a novel programming model and library API for developing portable parallel programs on heterogeneous hybrid servers composed of a multicore CPU and one or more different types of accelerators. OpenH integrates Pthreads, OpenMP, and OpenACC seamlessly to facilitate the development of hybrid parallel programs. An OpenH hybrid parallel program starts as a single main thread, creating a group of Pthreads called hosting Pthreads. A hosting Pthread then leads the execution of a software component of the program, either an OpenMP multithreaded component running on the CPU cores or an OpenACC (or OpenMP) component running on one of the accelerators of the server. The OpenH library provides API functions that allow programmers to get the configuration of the executing environment and bind the hosting Pthreads (and hence the execution of components) of the program to the CPU cores of the hybrid server to get the best performance. We illustrate the OpenH programming model and library API using two hybrid parallel applications based on matrix multiplication and 2D fast Fourier transform for the most general case of a hybrid hyperthreaded server comprising computing devices. Finally, we demonstrate the practical performance and energy consumption of OpenH for the hybrid parallel matrix multiplication application on a server comprising an Intel Icelake multicore CPU and two Nvidia A40 GPUs.

Generalizing Hierarchical Parallelism

NestedMP: Taming Complex Configuration Space of Degree of Parallelism for Nested-Parallel Programs

A New Parallel Skeleton for General Accumulative Computations

Extending the Nested Parallel Model to the Nested Dataflow Model with Provably Efficient Schedulers

Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning

Towards Polytypic Parallel Programming

A Uniform Approach Toward Nested Parallelism

Towards whatever-scale abstractions for data-driven parallelism

NestedMP: Enabling Cache-Aware Thread Mapping for Nested Parallel Shared Memory Applications

An Accumulative Parallel Skeleton for All

Parallel Programming Models and Languages

The best multicore-parallelization refactoring you've never heard of

Parallelization with Tree Skeletons

HeteroPP: A directive‐based heterogeneous cooperative parallel programming framework

Enhanced Parallelization via Constraints

Calculational Parallel Programming (Parallel Programming With Homomorphism And Mapreduce)

Models of parallel computation: a survey and classification

Hybrid Parallel Programming Model for Hierarchical NoC

Hierarchical Bulk Synchronous Parallel Model and Performance Optimization

OpenH: A Novel Programming Model and API for Developing Portable Parallel Programs on Heterogeneous Hybrid Servers