OS kernel supported fault tolerant MPI

Yingming Chen,Zhihui Du,Lin Peng,Sanli Li

2001-01-01

Abstract:Currently parallel cluster systems have a large number of computing nodes, and have a high risk of individual node failure. TH-MPI is a Linux kernel supported fault tolerant MPI. It optimizes the checkpointing by using Linux kernel module technology and diskless checkpointing. The results show good performance of checkpointing in TH-MPI.

What problem does this paper attempt to address?

TH-MPI: OS Kernel Integrated Fault Tolerant MPI.

Yu Chen,Qian Fang,Zhihui Du,Sanli Li

DOI: https://doi.org/10.1007/3-540-45417-9_15

2001-01-01

Abstract:Consisting of large numbers of computing nodes, parallel cluster systems have high risks of individual node failure. To overcome the high overhead drawbacks of current fault tolerant MPI systems, this paper presents TH-MPI for parallel cluster systems. Being integrated into Linux kernel, THMPI is implemented in a more effective, transparent and extensive way. With supports of dynamic kernel module and diskless checkpointing technologies, our experiment shows that checkpointing in TH-MPI is effectively optimized.
Checkpointing and Migration of parallel processes based on Message Passing Interface

Zhang Youhui,Wang Dongsheng,Zheng Weimin

2002-01-01

Abstract:This paper presents a Checkpoint-based Rollback Recovery and Migration System for Message Passing Interface, ChaRM4MPI, for Linux Clusters. Some important fault tolerant mechanisms are designed and implemented in this system, which include coordinated checkpointing protocol, synchronized rollback recovery, process migration, and so on. Owing to ChaRM4MPI, the node transient faults can be recovered automatically, and the permanent fault can also be recovered through checkpoint mirroring and process migration techniques. Moreover, users can migrate MPI processes from one node to another manually for load balance or system maintenance. ChaRM4MPI is a user-transparent implementation and introduces a little running time overhead.
Proposal of MPI Operation Level Checkpoint/Rollback and One Implementation

Yuan Tang,Graham E. Fagg,Jack J. Dongarra

DOI: https://doi.org/10.1109/ccgrid.2006.81

2006-01-01

Abstract:With the increasing number of processors in modern HPC(High Performance Computing) systems, there are two emergent problems to solve. One is scalability, the other is fault tolerance. In our previous work, we extended the MPI specification on handling fault tolerance by specifying a systematic framework for the recovery methods, communicator, message modes etc. that define the behavior of MPI in case an error occurs. These extensions not only specify how the implementation of the MPI library and RTE (Run Time Environment) handle failures at the system level, but provide the normal HPC application developers with various recovery choices with varying performance and cost. In this paper, we continue the work on extending the MPI's capability in this direction. Firstly, we are proposing an MPI operation level checkpoint/rollback library to recover the user's data. More importantly, we argue that the future generation programming model of a fault tolerant MPI application should be recover-and-continue against the more traditional stop-and-restart model. Recover-and-continue means that in case an error occurs, we just re-spawn the failed processes. All the remaining living processes stay in their original processors mapping on memory. The main benefits of recover-and-continue are much less cost for system recovery and the opportunity of employing in-memory checkpoint/ rollback techniques. Compared with stable or local disk techniques, which are the only choices for stop-andrestart, doubtlessly, the in-memory approach significantly reduces the performance penalty in checkpoint/rollback. Additionally, it makes it possible to establish a concurrent multiple level checkpoint/ rollback framework. With the progress of our work, a picture of the hierarchy of future generation fault tolerant HPC system will be gradually unveiled.
Building algorithmically nonstop fault tolerant MPI programs

Rui Wang,Erlin Yao,Mingyu Chen,Guangming Tan,Pavan Balaji,Darius Buntinas

DOI: https://doi.org/10.1109/HiPC.2011.6152716

2011-01-01

Abstract:With the growing scale of high-performance computing (HPC) systems, today and more so tomorrow, faults are a norm rather than an exception. HPC applications typically tolerate fail-stop failures under the stop-and-wait scheme, where even if only one processor fails, the whole system has to stop and wait for the recovery of the corrupted data. It is now a more-or-less accepted fact that the stop-and-wait scheme will not scale to the next generation of HPC systems. Inspired by the previous stop-and-wait algorithm-based fault tolerance (ABFT) recovery technique, we propose in this paper a nonstop fault tolerance scheme at the application level and describe its implementation. When failure occurs during the execution of applications, we do not stop to wait for the recovery of the corrupted node; instead, we replace it with the corresponding redundant node and continue the execution. At the end of execution, the correct solution can be recovered algorithmically at a very low cost. In order to implement the scheme, some new fault-tolerant features of the Message Passing Interface (MPI) have been investigated and utilized in the MPICH implementation of MPI. We also describe a case study using High Performance Linpack (HPL) with these new features and evaluate the performance of both our new scheme and ABFT recovery. Experimental results show the advantage of our new scheme over ABFT recovery even in a small scale.
A checkpointing/recovery system for MPI applications on cluster of IA-64 computers

Youhui Zhang,Ruini Xue,Dongsheng Wong,Weimin Zheng

DOI: https://doi.org/10.1109/ICPPW.2005.5

2005-01-01

Abstract:As the clusters continue to grow in size and popularity, issues of fault tolerance and reliability turn into limiting factors on application scalability and system availability. To address these issues, we design and implement a high availability parallel run-time system - ChaRM64 for MPI, a checkpoint-based rollback recovery and migration system for MPI programs on a cluster of IA-64 computers. Our approach integrates MPICH with a user-level, single process checkpoint/recovery library for IA-64 Linux, and modifies P4 libraries to implement a coordinated checkpointing and rollback recovery (CRR) and migration mechanism for parallel applications. In addition, the CRR of file operations is supported. Testing shows negligible performance overhead introduced by the CRR mechanism in our implementation.
Fault-Tolerant Clusters of Workstations with Single System Image

Kai Hwang,Edward Chow,Cho-Li Wang,Hai Jin,Zhiwei Xu

1998-01-01

Cluster Computing

Abstract:he computing trend is moving from clustering highend mainframes to clustering desktop computers. This trend is triggered by the widespread use of PCs, workstations, Gigabit networks, and middleware support for clustering. This paper presents new approaches to achieving fault tolerance and single system image (SSI) in a workstation cluster. A multicomputer cluster is a collection of node computers, which are physically connected by local area networks or high-bandwidth switch networks using optical fibres. The workstations in the cluster can work collectively as an integrated computing resource, that is a SSI, or they can operate as individual computers, separately.
User-level Checkpoint and Recovery for LAM/MPI

Youhui Zhang,Dongsheng Wong,Weimin Zheng

DOI: https://doi.org/10.1145/1075395.1075402

2005-01-01

Abstract:As high-performance clusters continue to grow in size and popularity, issues of fault tolerance and reliability are becoming limiting factors on application scalability. We integrated one user-level checkpointing and rollback recovery (CRR) library to LAM/MPI, a high performance implementation of the Message Passing Interface (MPI), to improve its availability. Compared with the current CRR implementation of LAM/MPI, our work supports file checkpointing and own higher portability, which can run on more platforms including IA32 and IA64 Linux. In addition, the test shows that less than 15% performance overhead is introduced by the CRR mechanism of our implementation.
FT-Offload: A Scalable Fault-Tolerance Programing Model on MIC Cluster

Cheng Chen,Yunfei Du,Zhen Xu,Canqun Yang

DOI: https://doi.org/10.1007/978-3-319-27140-8_1

2015-01-01

Abstract:Massively heterogeneous architectures are popular for modern petascale and future exascale systems. Fault-tolerance is key to the increased number of components and the complexity of these heterogeneous systems. However, standard offload programming models have traditionally been developed for supporting high performance rather than reliability. Naive fault-tolerance protocols are incapable of serving distributed MPI applications that tuned for CPU-MIC heterogeneous clusters. To address these problems, we design and implement a framework of fault tolerance programming model (FT-Offload). This enhances the reliability of heterogeneous supercomputers and retains the convenient of popular Intel Offload programming model. The effectiveness of the framework is demonstrated via numerical analysis and by porting both benchmarks and real-world applications to large-scale CPU-MIC nodes on the Tianhe-2 supercomputer. Our experimental results show that the current solution, which involves checkpoints, can efficiently strength the long running and reduce checkpointing overhead.
Utilizing the Multi-threading Techniques to Improve the Two-Level Checkpoint/Rollback System for MPI Applications

Yuan Tang,Yunquan Zhan

DOI: https://doi.org/10.1109/hpcc.2008.58

2008-01-01

Abstract:With the increasing number of processors in modern HPC (High Performance Computing) systems, there are two emergent problems to solve. One is scalability, the other is fault tolerance. In our previous work, we proposed an MPI operation level checkpoint/rollback system. The main benefits of the system is that it offers the opportunity to employ in-memory (disk-less) checkpoint/rollback techniques which has demonstrated a much better performance over its on-disk counterpart, and the opportunity to have a concurrent two level recover-and-continue MPI system which has been proven to have a high efficiency. To the scope of my knowledge, this is the first concurrent two-level checkpoint/recovery system in use. With the coming of Multi-core era, it's time to utilize the Multi-threadingtechniques to improve the performance of in-memory checkpointing algorithm. In this paper, we present two versions of MPI operation level checkpoint/rollback system, one is of single-threaded, the other is of Multi-threaded. Also, we provide an in-depth performance analysis between these two approaches to illustrate the benefits of Multi-threading techniques on multi-core platform. With the progress of our work, a picture of the hierarchy of future generation fault tolerant HPC system is gradually unrolled.
Study of Multi-nodes Failover Used in HA Cluster

WANG Ming-wei,YIN Kang-kai,LI Shan-ping

DOI: https://doi.org/10.3969/j.issn.1001-3695.2005.03.026

2005-01-01

Abstract:Multi-nodes model is introduced. The test results show that the failover technology works well and can quickly and exactly remove the tasks of any invalid working node to the copied one to keep the clustering service continuous.
MATCH: An MPI Fault Tolerance Benchmark Suite

Luanzheng Guo,Giorgis Georgakoudis,Konstantinos Parasyris,Ignacio Laguna,Dong Li

DOI: https://doi.org/10.48550/arXiv.2102.06894

2021-02-13

Abstract:MPI has been ubiquitously deployed in flagship HPC systems aiming to accelerate distributed scientific applications running on tens of hundreds of processes and compute nodes. Maintaining the correctness and integrity of MPI application execution is critical, especially for safety-critical scientific applications. Therefore, a collection of effective MPI fault tolerance techniques have been proposed to enable MPI application execution to efficiently resume from system failures. However, there is no structured way to study and compare different MPI fault tolerance designs, so to guide the selection and development of efficient MPI fault tolerance techniques for distinct scenarios. To solve this problem, we design, develop, and evaluate a benchmark suite called MATCH to characterize, research, and comprehensively compare different combinations and configurations of MPI fault tolerance designs. Our investigation derives useful findings: (1) Reinit recovery in general performs better than ULFM recovery; (2) Reinit recovery is independent of the scaling size and the input problem size, whereas ULFM recovery is not; (3) Using Reinit recovery with FTI checkpointing is a highly efficient fault tolerance design. MATCH code is available at <a class="link-external link-https" href="https://github.com/kakulo/MPI-" rel="external noopener nofollow">this https URL</a> FT- Bench.

Distributed, Parallel, and Cluster Computing
Fault-tolerant Mechanism of the Distributed Cluster Computers

Shang Yizi,Jin Yang,Wu Baosheng

DOI: https://doi.org/10.1016/s1007-0214(07)70107-4

2007-01-01

Abstract:The distributed system with high performance and stability is commonly adopted in large scale scientific and engineering computing. In this paper, we discuss a fault-tolerant mechanism under Linux circumstance to improve the fault-tolerant ability of the system, namely a scheme and frame to form the stable computing platform. In terms of the structure and function of the distributed system, active list and file invocation strategies are employed in the task management. System multilevel fault-tolerance can be achieved by repeated processes in a single node and task migration on multi-nodes. Manager node agent introduced in this paper administrates the nodes using the list, disposes of the tasks according to the nodes' performance, and hence, to be able to make full use of the cluster resources. An evaluation method is proposed to appraise the performance. The analyzed results show the usefulness of the scheme proposed except for some additional overhead of memory consumption.
Self-Checkpoint

Xiongchao Tang,Jidong Zhai,Bowen Yu,Wenguang Chen,Weimin Zheng

DOI: https://doi.org/10.1145/3155284.3018745

2017-01-01

Abstract:Fault tolerance is increasingly important in high performance computing due to the substantial growth of system scale and decreasing system reliability. In-memory/diskless checkpoint has gained extensive attention as a solution to avoid the IO bottleneck of traditional disk-based checkpoint methods. However, applications using previous in-memory checkpoint suffer from little available memory space. To provide high reliability, previous in-memory checkpoint methods either need to keep two copies of checkpoints to tolerate failures while updating old checkpoints or trade performance for space by flushing in-memory checkpoints into disk. In this paper, we propose a novel in-memory checkpoint method, called self-checkpoint, which can not only achieve the same reliability of previous in-memory checkpoint methods, but also increase the available memory space for applications by almost 50%. To validate our method, we apply the self-checkpoint to an important problem, fault tolerant HPL. We implement a scalable and fault tolerant HPL based on this new method, called SKT-HPL, and validate it on two large-scale systems. Experimental results with 24,576 processes show that SKT-HPL achieves over 95% of the performance of the original HPL. Compared to the state-of-the-art in-memory checkpoint method, it improves the available memory size by 47% and the performance by 5%.
Parallel Checkpoint/Recovery on Cluster of IA-64 Computers

Youhui Zhang,Dongsheng Wang,Weimin Zheng

DOI: https://doi.org/10.1007/978-3-540-30566-8_26

2004-01-01

Abstract:We design and implement a high availability parallel run-time system—ChaRM64, a Checkpoint- based Rollback Recovery and Migration system for parallel running programs on a cluster of IA-64 computers. At first, we discuss our solution of a user-level, single process checkpoint/recovery library running on IA-64 systems. Based on this library, ChaRM64 is realized, which implements a user-transparent, coordinated checkpointing and rollback recovery (CRR) mechanism, quasi-asynchronous migration and the dynamic reconfiguration function. Owing to the above techniques and efficient error detection, ChaRM64 can handle cluster node crashes and hardware transient faults in a IA-64 cluster. Now ChaRM64 for PVM has been implemented in Linux and the MPI version is under construction. As we know, there are few similar projects accomplished for IA-64 architecture.
Building a Fault Tolerant Application Using the GASPI Communication Layer

Faisal Shahzad,Moritz Kreutzer,Thomas Zeiser,Rui Machado,Andreas Pieper,Georg Hager,Gerhard Wellein

DOI: https://doi.org/10.1109/cluster.2015.106

2015-09-01

Abstract:It is commonly agreed that highly parallel software on Exascale computers will suffer from many more runtime failures due to the decreasing trend in the mean time to failures (MTTF). Therefore, it is not surprising that a lot of research is going on in the area of fault tolerance and fault mitigation. Applications should survive a failure and/or be able to recover with minimal cost. MPI is not yet very mature in handling failures, the User-Level Failure Mitigation (ULFM) proposal being currently the most promising approach is still in its prototype phase. In our work we use GASPI, which is a relatively new communication library based on the PGAS model. It provides the missing features to allow the design of fault-tolerant applications. Instead of introducing algorithm-based fault tolerance in its true sense, we demonstrate how we can build on (existing) clever checkpointing and extend applications to allow integrate a low cost fault detection mechanism and, if necessary, recover the application on the fly. The aspects of process management, the restoration of groups and the recovery mechanism is presented in detail. We use a sparse matrix vector multiplication based application to perform the analysis of the overhead introduced by such modifications. Our fault detection mechanism causes no overhead in failure-free cases, whereas in case of failure(s), the failure detection and recovery cost is of reasonably acceptable order and shows good scalability.
A faster checkpointing and recovery algorithm with a hierarchical storage approach

Wen GAO,Mingyu CHEN,Takashi NANYA

DOI: https://doi.org/10.1109/HPCASIA.2005.2

2005-01-01

Abstract:Fault tolerance is an inevitable part of cluster operating system. In Score cluster system, it provides coordinated checkpointing, rollback recovery mechanism and watch-dog timer detector for fault tolerance. In the checkpointing algorithm in Score, disk write is the bottleneck. To eliminate disk write overhead, this paper proposes a new diskless checkpointing and rollback recovery algorithm. Since the proposed algorithm does not need to calculate parity and write the checkpointing data into disk, it is analyzed to be a faster checkpointing algorithm than the original one. Based on comparison, the recovery time of the proposed algorithm is also less. However, the cluster can not tolerant multiple transient failure using this diskless checkpointing algorithm. To compensate this drawback, a hierarchical storage strategy is adopted. An experimental result shows that this diskless algorithm with a hierarchical storage approach is fast and effective
Research on Fault Tolerance in Hybrid P2P-based Collaborative Systems

Hao Fu,Ming Cai,Wenmin Zhu,Zhongdong Huang,Jinxiang Dong

DOI: https://doi.org/10.1109/CSCWD.2007.4281405

2007-01-01

Abstract:Hybrid P2P technique recently has become more and more popular for collaborative systems. However, reliability is issue of major importance in the server end of these systems. TMR technique for Collaborative System (CS-TMR) is presented in this paper to improve the server reliability. This software schema incorporates three homogeneous microcomputers and provides the fault-tolerant function through package interfaces to server applications. As it is COS-based, the method is more general-purpose, and programmers need not pay too much attention to the fault tolerance technology in detail. This method helps the collaborative server work in normal and degraded (duple or even single modular) modes, and can tolerate transient or permanent faults. Meanwhile, due to the importance of the server in hybrid P2P server, server application upgrade is impossible without server stop. A novel seamless software upgrade method is brought forward through the intelligent state-transition-control in CS-TMR package to reduce the cost of software upgrade.
Checkpoint-Restart Libraries Must Become More Fault Tolerant

Anthony Skjellum,Derek Schafer

DOI: https://doi.org/10.48550/arXiv.2112.10814

2021-12-21

Abstract:Production MPI codes need checkpoint-restart (CPR) support. Clearly, checkpoint-restart libraries must be fault tolerant lest they open up a window of vulnerability for failures with byzantine outcomes. But, certain popular libraries that leverage MPI are evidently not fault tolerant. Nowadays, fault detection with automatic recovery without batch requeueing is a strong requirement for production environments. Thus, allowing deadlock and setting long timeouts are suboptimal for fault detection even when paired with conservative recovery from the penultimate checkpoint. When MPI is used as a communication mechanism within a CPR library, such libraries must offer fault-tolerant extensions with minimal detection, isolation, mitigation, and potential recovery semantics to aid the CPR's library fail-backward. Communication between MPI and the checkpoint library regarding system health may be valuable. For fault-tolerant MPI programs (e.g., using APIs like FA-MPI, Stages/Reinit, or ULFM), the checkpoint library must cooperate with the extended model or else invalidate fault-tolerant operation.

Distributed, Parallel, and Cluster Computing
A New and Efficient Algorithm-Based Fault Tolerance Scheme for A Million Way Parallelism

Erlin Yao,Mingyu Chen,Rui Wang,Wenli Zhang,Guangming Tan

DOI: https://doi.org/10.48550/arXiv.1106.4213

2011-06-21

Abstract:Fault tolerance overhead of high performance computing (HPC) applications is becoming critical to the efficient utilization of HPC systems at large scale. HPC applications typically tolerate fail-stop failures by checkpointing. Another promising method is in the algorithm level, called algorithmic recovery. These two methods can achieve high efficiency when the system scale is not very large, but will both lose their effectiveness when systems approach the scale of Exaflops, where the number of processors including in system is expected to achieve one million. This paper develops a new and efficient algorithm-based fault tolerance scheme for HPC applications. When failure occurs during the execution, we do not stop to wait for the recovery of corrupted data, but replace them with the corresponding redundant data and continue the execution. A background accelerated recovery method is also proposed to rebuild redundancy to tolerate multiple times of failures during the execution. To demonstrate the feasibility of our new scheme, we have incorporated it to the High Performance Linpack. Theoretical analysis demonstrates that our new fault tolerance scheme can still be effective even when the system scale achieves the Exaflops. Experiment using SiCortex SC5832 verifies the feasibility of the scheme, and indicates that the advantage of our scheme can be observable even in a small scale.

Distributed, Parallel, and Cluster Computing
What does fault tolerant Deep Learning need from MPI?

Vinay Amatya,Abhinav Vishnu,Charles Siegel,Jeff Daily

DOI: https://doi.org/10.48550/arXiv.1709.03316

2017-09-11

Abstract:Deep Learning (DL) algorithms have become the de facto Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive - even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults - requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: What is needed from MPI for de- signing fault tolerant DL implementations? In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by ex- tending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.

Distributed, Parallel, and Cluster Computing

OS kernel supported fault tolerant MPI

TH-MPI: OS Kernel Integrated Fault Tolerant MPI.

Checkpointing and Migration of parallel processes based on Message Passing Interface

Proposal of MPI Operation Level Checkpoint/Rollback and One Implementation

Building algorithmically nonstop fault tolerant MPI programs

A checkpointing/recovery system for MPI applications on cluster of IA-64 computers

Fault-Tolerant Clusters of Workstations with Single System Image

User-level Checkpoint and Recovery for LAM/MPI

FT-Offload: A Scalable Fault-Tolerance Programing Model on MIC Cluster

Utilizing the Multi-threading Techniques to Improve the Two-Level Checkpoint/Rollback System for MPI Applications

Study of Multi-nodes Failover Used in HA Cluster

MATCH: An MPI Fault Tolerance Benchmark Suite

Fault-tolerant Mechanism of the Distributed Cluster Computers

Self-Checkpoint

Parallel Checkpoint/Recovery on Cluster of IA-64 Computers

Building a Fault Tolerant Application Using the GASPI Communication Layer

A faster checkpointing and recovery algorithm with a hierarchical storage approach

Research on Fault Tolerance in Hybrid P2P-based Collaborative Systems

Checkpoint-Restart Libraries Must Become More Fault Tolerant

A New and Efficient Algorithm-Based Fault Tolerance Scheme for A Million Way Parallelism

What does fault tolerant Deep Learning need from MPI?