Abstract:Over the past two decades, distributed cluster system has evolved from none to the predominant architecture in the current HPC world — it accounts for about 85% of the current top500 supercomputers. Along with the rise of cluster, MPI has evolved into the de facto standard for HPC applications on distributed clusters due to its great portability and performance. The tool support for verifying correctness of MPI applications lags far behind the ever-increasingly sophisticated hardware that has been developed. Lack of tools poses following challenges for developers: (1) there are no practical tools for systematic software testing techniques for MPI applications; and (2) there is insufficient tool support for scaling problems — a class of bugs that manifests at large scale either in terms of problem size or the number of processes. To address the first problem, we have developed COMPI, the first concolic testing tool for MPI applications. COMPI tackles two major challenges. First, it provides an automated testing framework for MPI programs — it performs concolic execution on a single process and records branch coverage across all. Second, COMPI employs three techniques to effectively control the cost of testing as too high a cost may prevent its adoption or even make the testing infeasible. To address the second problem, we have designed an avoidance framework for scaling problems with the use of MPI collectives. As ∗The original title in the list of IPDPS’18 PhD Forum is Correctness Support for MPI Applications. the complexity of MPI collectives is directly impacted by both parallelism scale and problem size, their use often triggers scaling problems. Fixing a scaling problem is challenging, and thus it usually takes much time for users to wait for an official fix. To improve users’ productivity, we establish the necessity of user side testing and provide a protection layer to avoid scaling problems non-intrusively, i.e., without requiring any changes to the MPI library or user programs. This provides an immediate remedy when an official fix is not readily available. We also built a hang detection tool that saves computing resources in presence of program hangs at large scale. While program hangs on large parallel systems can be detected via the widely used timeout mechanism, it is difficult for the users to set the timeout – too small a timeout leads to high false alarm rates and too large a timeout wastes a vast amount of valuable computing resources. To address the above problems with hang detection, this paper presents ParaStack, an extremely lightweight tool to detect hangs in a timely manner with high accuracy, negligible overhead with great scalability, and without requiring the user to select a timeout value. For a detected hang, it provides direction for further analysis by telling users whether the hang is the result of an error in the computation phase or the communication phase. For a computation-error induced hang, our tool pinpoints the faulty process by excluding hundreds and thousands of other processes.

Non-intrusively Avoiding Scaling Problems in and out of MPI Collectives

Testing and Runtime Support for MPI Applications ∗

Optimizing MPI Collectives on Shared Memory Multi-Cores

A More Scalable Sparse Dynamic Data Exchange

COMPI: Concolic Testing for MPI Applications

Solution to Codes Scale Reduction of MPI and Implemention

Effective method of collective communication for message passing on cluster

A Survey of Potential MPI Complex Collectives: Large-Scale Mining and Analysis of HPC Applications

Optimizing Irregular Communication with Neighborhood Collectives and Locality-Aware Parallelism

Efficient concolic testing of MPI applications

An Optimized Error-controlled MPI Collective Framework Integrated with Lossy Compression

Enabling Highly-Scalable Remote Memory Access Programming with MPI-3 One Sided

Building algorithmically nonstop fault tolerant MPI programs

Proposal of MPI Operation Level Checkpoint/Rollback and One Implementation

ScaleCheck: A Single-Machine Approach for Discovering Scalability Bugs in Large Distributed Systems

To parallelize or not to parallelize, bugs issue

Study on MPI/OpenMP hybrid parallelism for Monte Carlo neutron transport code

MPI+Threads: runtime contention and remedies

Identifying Scalability Bottlenecks for Large-Scale Parallel Programs with Graph Analysis

Increasing the Efficiency of Sparse Matrix-Matrix Multiplication with a 2.5D Algorithm and One-Sided MPI

Decomposing Collectives for Exploiting Multi-lane Communication