Abstract:Distributed computing is the mainstream for large-scale machine learning and big data analytics, but its performance usually suffers from unpredictable stragglers, i.e., very slow nodes. Recently, coded computing has emerged as a new distributed computing paradigm that uses coding-theoretical approaches to mitigate the effect of stragglers. Most existing coding schemes only use the results from a certain number of fastest worker nodes to recover the output and completely ignore the partial work done by other worker nodes, leading to inferior performance. In this paper, for scenarios where each worker node transmits its local result to the master node only after it has finished the whole local computation, we introduce communication at full speed to characterize the full utilization of all the communication links between each worker node that has finished its local computation and the master node, and for scenarios where each worker node computes out the local result piece by piece and can forward each piece once available, we introduce computation at full speed to characterize the full utilization of the work done entirely or partially by all the worker nodes. Considering a general polynomial-based coding framework which encapsulates many advanced coding schemes for a variety of fundamental computing tasks, we propose a randomized approach where each worker node partitions its local result into pieces, generates and forwards random linear combinations of these pieces to the master node sequentially, and theoretically demonstrate that it can lead the coding framework to achieve communication at full speed. For some typical task scenarios, we further show that computation at full speed can be achieved by mapping the encoding operations in the randomized approach into a part of encoding on the input dataset. Experiments conducted on Alibaba Cloud as well as simulations show that our approaches can reduce the total runtime significantly.

Block-Division-Based Wireless Coded Computation

Coded Computing at Full Speed

Network Coding Approaches for Distributed Computation over Lossy Wireless Networks.

A Survey of Coded Distributed Computing

Coded Computation across Shared Heterogeneous Workers with Communication Delay

Delay Optimization of Coded Computing with Hierarchical Task Partitioning

Task-Decoding Assisted Cooperative Transmission for Coded Edge Computing

Hybrid Coded MapReduce for Latency-Constrained Tasks with Straggling Servers

On the Capacity of Privacy-Preserving and Straggler-Robust Distributed Coded Computing

Flow-oriented Network Coding Architecture for Multihop Wireless Networks.

Wireless MapReduce Arrays for Coded Distributed Computing

Adaptive Privacy-Preserving Coded Computing with Hierarchical Task Partitioning

A General Packet Coding Scheme For Multi-Hop Wireless Networks

Compressed Coded Distributed Computing

A New Coding Scheme for Matrix-Vector Multiplication Via Universal Decodable Matrices.

Heterogeneous Coded Computation Across Heterogeneous Workers.

Breaking Blockchain's Communication Barrier with Coded Computation

An Efficient Chunked Network Code Based Transmission Scheme in Wireless Networks

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

How to Optimally Allocate Resources for Coded Distributed Computing?