Fundamental Limits of Distributed Linearly Separable Computation under Cyclic Assignment

Wenbo Huang,Kai Wan,Hua Sun,Mingyue Ji,Robert Caiming Qiu,Giuseppe Caire

2024-02-19

Abstract:This paper studies the master-worker distributed linearly separable computation problem, where the considered computation task, referred to as linearly separable function, is a typical linear transform model widely used in cooperative distributed gradient coding, real-time rendering, linear transformers, etc. %A master asks $\Nsf$ distributed workers to compute a linearly separable function from $\Ksf$ datasets. The computation task on $\Ksf$ datasets can be expressed as $\Ksf_{\rm c}$ linear combinations of $\Ksf$ messages, where each message is the output of an individual function on one dataset. Straggler effect is also considered, such that from the answers of any $\Nsf_{\rm r}$ of the $\Nsf$ distributed workers, the master should accomplish the task. The computation cost is defined as the number of datasets assigned to each worker, while the communication cost is defined as the number of (coded) messages that should be received. The objective is to characterize the optimal tradeoff between the computation and communication costs. The problem has remained so far open, even under the cyclic data assignment.Since in fact various distributed computing schemes were proposed in the literature under the cyclic data assignment, with this paper we close the problem for the cyclic assignment. This paper proposes a new computing scheme with the cyclic assignment based on the concept of interference alignment, by treating each message which cannot be computed by a worker as an interference from this worker. Under the cyclic assignment, the proposed computing scheme is then proved to be optimal when $\Nsf=\Ksf$ and be order optimal within a factor of $2$ otherwise.

Information Theory

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to optimize the trade - off between computational cost and communication cost in distributed linearly separable computing while considering the "dragging - effect" (i.e., some worker nodes are slow or fail in computing). Specifically, the paper studies the master - slave distributed linearly separable computing problem, where the computing task can be represented as \(K\) linear combinations on \(K\) data sets, and each linear combination corresponds to the output of a function on a single data set. The goal is to minimize the number of data sets assigned to each worker node (computational cost) and the number of messages to be received (communication cost) under the premise of ensuring the completion of the task.

The main contribution of the paper lies in proposing a new computing scheme, which is based on the interference alignment strategy and achieves an optimal or near - optimal computational - communication cost trade - off under cyclic allocation conditions. In particular, when \(N = K\), the proposed scheme is proven to be completely optimal; while in other cases, the performance of the scheme is also within twice the optimal solution. In addition, experimental results show that this scheme is significantly superior to existing methods in practical applications. For example, tests on Tencent Cloud show that the time of the entire distributed computing process is reduced by up to 72.8%.