Abstract:The International Journal of High Performance Computing Applications, Ahead of Print. As HPC system architectures and the applications running on them continue to evolve, the MPI standard itself must evolve. The trend in current and future HPC systems toward powerful nodes with multiple CPU cores and multiple GPU accelerators makes efficient support for hybrid programming critical for applications to achieve high performance. However, the support for hybrid programming in the MPI standard has not kept up with recent trends. The MPICH implementation of MPI provides a platform for implementing and experimenting with new proposals and extensions to fill this gap and to gain valuable experience and feedback before the MPI Forum can consider them for standardization. In this work, we detail six extensions implemented in MPICH to increase MPI interoperability with other runtimes, with a specific focus on heterogeneous architectures. First, the extension to MPI generalized requests lets applications integrate asynchronous tasks into MPI's progress engine. Second, the iovec extension to datatypes lets applications use MPI datatypes as a general-purpose data layout API beyond just MPI communications. Third, a new MPI object, MPIX_Stream, can be used by applications to identify execution contexts beyond MPI processes, including threads and GPU streams. MPIX stream communicators can be created to make existing MPI functions thread-aware and GPU-aware, thus providing applications with explicit ways to achieve higher performance. Fourth, MPIX Streams are extended to support the enqueue semantics for offloading MPI communications onto a GPU stream context. Fifth, thread communicators allow MPI communicators to be constructed with individual threads, thus providing a new level of interoperability between MPI and on-node runtimes such as OpenMP. Lastly, we present an extension to invoke MPI progress, which lets users spawn progress threads with fine-grained control to adapt the communication performance to their application designs. We describe the design and implementation of these extensions, provide usage examples, and highlight their expected benefits with performance results.

Parallel Programming with MatlabMPI

Survey of Parallel Computing with MATLAB

DataMPI: Extending MPI to Hadoop-Like Big Data Computing

Accelerating Apache Hive with MPI for Data Warehouse Systems

Parallel MATLAB Techniques

Distributed Parallel Optimization Computation Environment Based on Matlab

Parallel Multi-Scale Computation Using The Message Passing Interface

MPI Implementation Profiling for Better Application Performance

Enabling MPI communication within Numba/LLVM JIT-compiled Python code using numba-mpi v1.0

Parallel Implementations of the Jacobi Linear Algebraic Systems Solve

Parallel Computing of Shared Memory Multiprocessors Based on JOMP

Designing and prototyping extensions to the Message Passing Interface in MPICH

Parallel Power Flow Algorithm Based on MPI and P-Q Analysis Method

The parallelization of SPIDER on distributed-memory computers using MPI.

Message-Passing Computing with Java: Performance Evaluation and Comparisons.

Parallelization of Mcnp Monte Carlo Neutron and Photon Transport Code in Parallel Virtual Machine and Message Passing Interface

Interfacing Interpreted and Compiled Languages to Support Applications on a Massively Parallel Network of Workstations (MP-NOW)

Parallel Computing for Fringe Pattern Processing: A Multicore CPU Approach in MATLAB® Environment

MPICH-G2: A Grid-Enabled Implementation of the Message Passing Interface

A Parallel Algorithm Framework for Feature Extraction of EEG Signals on MPI.

MPIrigen: MPI Code Generation through Domain-Specific Language Models