Abstract:Existing checkpointing approaches seem ill-suited for distributed training even though hardware limitations make model parallelism, i.e., sharding model state across multiple accelerators, a requirement for model scaling. Consolidating distributed model state into a single checkpoint unacceptably slows down training, and is impractical at extreme scales. Distributed checkpoints, in contrast, are tightly coupled to the model parallelism and hardware configurations of the training run, and thus unusable on different configurations. To address this problem, we propose Universal Checkpointing, a technique that enables efficient checkpoint creation while providing the flexibility of resuming on arbitrary parallelism strategy and hardware configurations. Universal Checkpointing unlocks unprecedented capabilities for large-scale training such as improved resilience to hardware failures through continued training on remaining healthy hardware, and reduced training time through opportunistic exploitation of elastic capacity. The key insight of Universal Checkpointing is the selection of the optimal representation in each phase of the checkpointing life cycle: distributed representation for saving, and consolidated representation for loading. This is achieved using two key mechanisms. First, the universal checkpoint format, which consists of a consolidated representation of each model parameter and metadata for mapping parameter fragments into training ranks of arbitrary model-parallelism configuration. Second, the universal checkpoint language, a simple but powerful specification language for converting distributed checkpoints into the universal checkpoint format. Our evaluation demonstrates the effectiveness and generality of Universal Checkpointing on state-of-the-art model architectures and a wide range of parallelism techniques.

CheckMate: Evaluating Checkpointing Protocols for Streaming Dataflows

Optimal Multi-Level Interval-based Checkpointing for Exascale Stream Processing Systems

A Utilization Model for Optimization of Checkpoint Intervals in Distributed Stream Processing Systems

Research on Optimal Checkpointing-Interval for Flink Stream Processing Applications

A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks

Non-Authentication Based Checkpoint Fault-tolerant Vulnerability in Spark Streaming

An efficient computing-checkpoint based coordinated checkpoint algorithm

A Study of Checkpointing in Large Scale Training of Deep Neural Networks

Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training

Performance Evaluation of an Algorithm-based Asynchronous Checkpoint-Restart Fault Tolerant Application Using Mixed MPI/GPI-2

Designing an Adaptive Application-Level Checkpoint Management System for Malleable MPI Applications

Realizing Best Checkpointing Control in Computing Systems

Benchmarking Distributed Stream Data Processing Systems

Enabling Practical Transparent Checkpointing for MPI: A Topological Sort Approach

On the Performance and Convergence of Distributed Stream Processing Via Approximate Fault Tolerance

A Cooperative Checkpointing Algorithm with Message Complexity O(n)

Energy-efficient localised rollback after failures via data flow analysis

Dynamic Resource Management In a Massively Parallel Stream Processing Engine

Delivery, consistency, and determinism: rethinking guarantees in distributed stream processing

A state lossless scheduling strategy in distributed stream computing systems

Low-Overhead Asynchronous Checkpointing in Main-Memory Database Systems