Abstract:Distributed computing is the mainstream for large-scale machine learning and big data analytics, but its performance usually suffers from unpredictable stragglers, i.e., very slow nodes. Recently, coded computing has emerged as a new distributed computing paradigm that uses coding-theoretical approaches to mitigate the effect of stragglers. Most existing coding schemes only use the results from a certain number of fastest worker nodes to recover the output and completely ignore the partial work done by other worker nodes, leading to inferior performance. In this paper, for scenarios where each worker node transmits its local result to the master node only after it has finished the whole local computation, we introduce communication at full speed to characterize the full utilization of all the communication links between each worker node that has finished its local computation and the master node, and for scenarios where each worker node computes out the local result piece by piece and can forward each piece once available, we introduce computation at full speed to characterize the full utilization of the work done entirely or partially by all the worker nodes. Considering a general polynomial-based coding framework which encapsulates many advanced coding schemes for a variety of fundamental computing tasks, we propose a randomized approach where each worker node partitions its local result into pieces, generates and forwards random linear combinations of these pieces to the master node sequentially, and theoretically demonstrate that it can lead the coding framework to achieve communication at full speed. For some typical task scenarios, we further show that computation at full speed can be achieved by mapping the encoding operations in the randomized approach into a part of encoding on the input dataset. Experiments conducted on Alibaba Cloud as well as simulations show that our approaches can reduce the total runtime significantly.

Communication-Efficient Coded Distributed Multi - Task Learning.

Coded Distributed Computing for Hierarchical Multi-task Learning

On Exploiting Network Topology for Hierarchical Coded Multi-task Learning

Distributed Mimo Network Optimization Based On Duality And Local Message Passing

Coded Parallelism for Distributed Deep Learning.

Heterogeneous Coded Computation Across Heterogeneous Workers.

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

Resource Allocation for Multi-Task Federated Learning Algorithm over Wireless Communication Networks

Compressed Coded Distributed Computing

HFedMTL: Hierarchical Federated Multi-Task Learning

Distributed Networked Multi-task Learning

Low-Latency Task-Oriented Communications with Multi-Round, Multi-Task Deep Learning

Distributed Multi-Task Relationship Learning

Federated Learning-Based Cooperative Model Training for Task-Oriented Semantic Communication

Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learning

Network Coding Approaches for Distributed Computation over Lossy Wireless Networks.

Joint Coding and Scheduling Optimization for Distributed Learning Over Wireless Edge Networks

UAV-Assisted Wireless Cooperative Communication and Coded Caching: A Multiagent Two-Timescale DRL Approach

Delay Optimization of Coded Computing with Hierarchical Task Partitioning

Coded Distributed Image Classification

Coded Computing at Full Speed