Abstract:Distributed computing is the mainstream for large-scale machine learning and big data analytics, but its performance usually suffers from unpredictable stragglers, i.e., very slow nodes. Recently, coded computing has emerged as a new distributed computing paradigm that uses coding-theoretical approaches to mitigate the effect of stragglers. Most existing coding schemes only use the results from a certain number of fastest worker nodes to recover the output and completely ignore the partial work done by other worker nodes, leading to inferior performance. In this paper, for scenarios where each worker node transmits its local result to the master node only after it has finished the whole local computation, we introduce communication at full speed to characterize the full utilization of all the communication links between each worker node that has finished its local computation and the master node, and for scenarios where each worker node computes out the local result piece by piece and can forward each piece once available, we introduce computation at full speed to characterize the full utilization of the work done entirely or partially by all the worker nodes. Considering a general polynomial-based coding framework which encapsulates many advanced coding schemes for a variety of fundamental computing tasks, we propose a randomized approach where each worker node partitions its local result into pieces, generates and forwards random linear combinations of these pieces to the master node sequentially, and theoretically demonstrate that it can lead the coding framework to achieve communication at full speed. For some typical task scenarios, we further show that computation at full speed can be achieved by mapping the encoding operations in the randomized approach into a part of encoding on the input dataset. Experiments conducted on Alibaba Cloud as well as simulations show that our approaches can reduce the total runtime significantly.

Group-wise Verifiable Coded Computing under Byzantine Attacks and Stragglers

Adaptive Verifiable Coded Computing: Towards Fast, Secure and Private Distributed Machine Learning

Verifiable Coded Computation of Multiple Functions

Generalized Lagrange Coded Computing: A Flexible Computation-Communication Tradeoff for Resilient, Secure, and Private Computation

Lagrange Coded Computing: Optimal Design For Resiliency, Security, And Privacy

Coded Computing for Resilient, Secure, and Privacy-Preserving Distributed Matrix Multiplication

Decode-and-Compare: An Efficient Verification Scheme for Coded Distributed Edge Computing

Interactive Byzantine-Resilient Gradient Coding for General Data Assignments

On the Capacity of Privacy-Preserving and Straggler-Robust Distributed Coded Computing

Download and Access Trade-offs in Lagrange Coded Computing.

Probabilistic Group Testing in Distributed Computing with Attacked Workers

Approximated Coded Computing: Towards Fast, Private and Secure Distributed Machine Learning

Securely Aggregated Coded Matrix Inversion

Coded Computation across Shared Heterogeneous Workers with Communication Delay

Trading Communication for Computation in Byzantine-Resilient Gradient Coding

Adaptive Privacy-Preserving Coded Computing With Hierarchical Task Partitioning

Coded Computing at Full Speed

One-Bit Byzantine-Tolerant Distributed Learning via Over-the-Air Computation

Coded Computing for Fault-Tolerant Parallel QR Decomposition