Abstract:With the increasing number of processors in modern HPC(High Performance Computing) systems, there are two emergent problems to solve. One is scalability, the other is fault tolerance. In our previous work, we extended the MPI specification on handling fault tolerance by specifying a systematic framework for the recovery methods, communicator, message modes etc. that define the behavior of MPI in case an error occurs. These extensions not only specify how the implementation of the MPI library and RTE (Run Time Environment) handle failures at the system level, but provide the normal HPC application developers with various recovery choices with varying performance and cost. In this paper, we continue the work on extending the MPI's capability in this direction. Firstly, we are proposing an MPI operation level checkpoint/rollback library to recover the user's data. More importantly, we argue that the future generation programming model of a fault tolerant MPI application should be recover-and-continue against the more traditional stop-and-restart model. Recover-and-continue means that in case an error occurs, we just re-spawn the failed processes. All the remaining living processes stay in their original processors mapping on memory. The main benefits of recover-and-continue are much less cost for system recovery and the opportunity of employing in-memory checkpoint/ rollback techniques. Compared with stable or local disk techniques, which are the only choices for stop-andrestart, doubtlessly, the in-memory approach significantly reduces the performance penalty in checkpoint/rollback. Additionally, it makes it possible to establish a concurrent multiple level checkpoint/ rollback framework. With the progress of our work, a picture of the hierarchy of future generation fault tolerant HPC system will be gradually unveiled.

Hardware rollback recovery schemes for multiprocessor systems

Fault Tolerant Real-Time Scheduling Strategy for NC System Based on Rollback Recovery

Error Recovery in a Real-Time Multiprocessor System

Rethread: A Low-Cost Transient Fault Recovery Scheme for Multithreaded Processors.

A New Operating System Fault Recovery Technique Based on Kernel Compensation and Process State Roll?Back

Rollback Recovery in Distributed Systems Using Loosely Synchronized Clocks

A Dual Process Redundancy Approach to Transient Fault Tolerance for Ccnuma Architecture

Fault-Tolerant Multitasking System Based on Interleaving of Threads

Hybrid Self Fault-tolerant Mechanism of Embedded Multi-tasks Software

HCRF: A Hardware Checkpoint-based Recovery Framework in Light Dual-Core Lockstep Processors

A high performance fault-tolerant approach based on simultaneous multithreading

A cycle-level recovery method for embedded processor against HT tamper

Efficient Scheduling Algorithm for Hard Real-Time Tasks in Primary-Backup Based Multiprocessor Systems

An efficient forward recovery checkpointing scheme in dissimilar redundancy computer system

Transient Fault Recovery on Chip Multiprocessor based on Dual Core Redundancy and Context Saving

Proposal of MPI Operation Level Checkpoint/Rollback and One Implementation

EnHTM: Exploiting Hardware Transaction Memory for Achieving Low-Cost Fault Tolerance

Efficient Fault-Tolerant Scheduling on Multiprocessor Systems Via Replication and Deallocation

Exploring Hardware Transaction Processing for Reliable Computing in Chip-Multiprocessors Against Soft Errors

Transient Fault Tolerance for Ccnuma Architecture

A New and Efficient Algorithm-Based Fault Tolerance Scheme for A Million Way Parallelism