Abstract:Distributed computing is the mainstream for large-scale machine learning and big data analytics, but its performance usually suffers from unpredictable stragglers, i.e., very slow nodes. Recently, coded computing has emerged as a new distributed computing paradigm that uses coding-theoretical approaches to mitigate the effect of stragglers. Most existing coding schemes only use the results from a certain number of fastest worker nodes to recover the output and completely ignore the partial work done by other worker nodes, leading to inferior performance. In this paper, for scenarios where each worker node transmits its local result to the master node only after it has finished the whole local computation, we introduce communication at full speed to characterize the full utilization of all the communication links between each worker node that has finished its local computation and the master node, and for scenarios where each worker node computes out the local result piece by piece and can forward each piece once available, we introduce computation at full speed to characterize the full utilization of the work done entirely or partially by all the worker nodes. Considering a general polynomial-based coding framework which encapsulates many advanced coding schemes for a variety of fundamental computing tasks, we propose a randomized approach where each worker node partitions its local result into pieces, generates and forwards random linear combinations of these pieces to the master node sequentially, and theoretically demonstrate that it can lead the coding framework to achieve communication at full speed. For some typical task scenarios, we further show that computation at full speed can be achieved by mapping the encoding operations in the randomized approach into a part of encoding on the input dataset. Experiments conducted on Alibaba Cloud as well as simulations show that our approaches can reduce the total runtime significantly.

Coded Computing for Multi-cluster Distributed Computations

Coded Distributed Computing with Heterogeneous Function Assignments

New Results on the Computation-Communication Tradeoff for Heterogeneous Coded Distributed Computing

Coded Distributed Computing: Fundamental Limits and Practical Challenges

Hybrid Coded MapReduce for Latency-Constrained Tasks with Straggling Servers

Multi-Access Distributed Computing

Compressed Coded Distributed Computing

How to Optimally Allocate Resources for Coded Distributed Computing?

Optimal Coding Scheme and Resource Allocation for Distributed Computation with Limited Resources

Cascaded Code Distributed Computing With Low Complexity and Improved Flexibility

Coded Computing for Half-Duplex Wireless Distributed Computing Systems Via Interference Alignment

Design and Optimization of Heterogeneous Coded Distributed Computing with Nonuniform File Popularity

Coded Wireless Distributed Computing Via Interference Alignment

Joint Design of Shuffling and Function Assignment in Heterogeneous Coded Distributed Computing

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Cascaded Coded Distributed Computing Schemes Based on Placement Delivery Arrays

On Implementation of Parallel Map and Shuffle Phases for Coded Distributed Computing

A Fundamental Tradeoff among Storage, Computation, and Communication for Distributed Computing over Star Network.

Coded Computing at Full Speed

Coded Computation across Shared Heterogeneous Workers with Communication Delay

Heterogeneous Coded Computation Across Heterogeneous Workers.