Abstract:With the increasing number of processors in modern HPC(High Performance Computing) systems, there are two emergent problems to solve. One is scalability, the other is fault tolerance. In our previous work, we extended the MPI specification on handling fault tolerance by specifying a systematic framework for the recovery methods, communicator, message modes etc. that define the behavior of MPI in case an error occurs. These extensions not only specify how the implementation of the MPI library and RTE (Run Time Environment) handle failures at the system level, but provide the normal HPC application developers with various recovery choices with varying performance and cost. In this paper, we continue the work on extending the MPI's capability in this direction. Firstly, we are proposing an MPI operation level checkpoint/rollback library to recover the user's data. More importantly, we argue that the future generation programming model of a fault tolerant MPI application should be recover-and-continue against the more traditional stop-and-restart model. Recover-and-continue means that in case an error occurs, we just re-spawn the failed processes. All the remaining living processes stay in their original processors mapping on memory. The main benefits of recover-and-continue are much less cost for system recovery and the opportunity of employing in-memory checkpoint/ rollback techniques. Compared with stable or local disk techniques, which are the only choices for stop-andrestart, doubtlessly, the in-memory approach significantly reduces the performance penalty in checkpoint/rollback. Additionally, it makes it possible to establish a concurrent multiple level checkpoint/ rollback framework. With the progress of our work, a picture of the hierarchy of future generation fault tolerant HPC system will be gradually unveiled.

FT-Offload: A Scalable Fault-Tolerance Programing Model on MIC Cluster

Building algorithmically nonstop fault tolerant MPI programs

Algorithmic Based Fault Tolerance Applied to High Performance Computing

A Case Study of Designing Efficient Algorithm-based Fault Tolerant Application for Exascale Parallelism

A New and Efficient Algorithm-Based Fault Tolerance Scheme for A Million Way Parallelism

Proposal of MPI Operation Level Checkpoint/Rollback and One Implementation

FT-GEMM: A Fault Tolerant High Performance GEMM Implementation on x86 CPUs

Design of COTS-based Fault-Tolerant Multiprocessor Real-Time Operating System

Research on Fault Tolerance in Hybrid P2P-based Collaborative Systems

TH-MPI: OS Kernel Integrated Fault Tolerant MPI.

Performance optimizations for scalable CFD applications on hybrid CPU+MIC heterogeneous computing system with millions of cores

Fault-tolerant Mechanism of the Distributed Cluster Computers

Model defined fault tolerance in cloud

Fault Tolerant One-Sided Matrix Decompositions on Heterogeneous Systems with GPUs.

A Case Study for Fault Tolerance Oriented Programming in Multi-core Architecture

OS kernel supported fault tolerant MPI

A Group-Based Fault Tolerant Mechanism For Heterogeneous Mobile Clouds

Fault-tolerant Design Techniques in A CMP Architecture

Cluster-level tuning of a shallow water equation solver on the Intel MIC architecture

Algorithm-Directed Crash Consistence in Non-Volatile Memory for HPC.

Implementing Performance Portability of High Performance Computing Programs in the New Golden Age of Chip Architecture