Abstract:We consider the communication complexity of a number of distributed optimization problems. We start with the problem of solving a linear system. Suppose there is a coordinator together with $s$ servers $P_1, \ldots, P_s$, the $i$-th of which holds a subset $A^{(i)} x = b^{(i)}$ of $n_i$ constraints of a linear system in $d$ variables, and the coordinator would like to output $x \in \mathbb{R}^d$ for which $A^{(i)} x = b^{(i)}$ for $i = 1, \ldots, s$. We assume each coefficient of each constraint is specified using $L$ bits. We first resolve the randomized and deterministic communication complexity in the point-to-point model of communication, showing it is $\tilde{\Theta}(d^2L + sd)$ and $\tilde{\Theta}(sd^2L)$, respectively. We obtain similar results for the blackboard model. When there is no solution to the linear system, a natural alternative is to find the solution minimizing the $\ell_p$ loss. While this problem has been studied, we give improved upper or lower bounds for every value of $p \ge 1$. One takeaway message is that sampling and sketching techniques, which are commonly used in earlier work on distributed optimization, are neither optimal in the dependence on $d$ nor on the dependence on the approximation $\epsilon$, thus motivating new techniques from optimization to solve these problems. Towards this end, we consider the communication complexity of optimization tasks which generalize linear systems. For linear programming, we first resolve the communication complexity when $d$ is constant, showing it is $\tilde{\Theta}(sL)$ in the point-to-point model. For general $d$ and in the point-to-point model, we show an $\tilde{O}(sd^3 L)$ upper bound and an $\tilde{\Omega}(d^2 L + sd)$ lower bound. We also show if one perturbs the coefficients randomly by numbers as small as $2^{-\Theta(L)}$, then the upper bound is $\tilde{O}(sd^2 L) + \textrm{poly}(dL)$.

Communication Complexity (for Algorithm Designers)

Communication Complexity and Its Applications

Classical Algorithms from Quantum and Arthur-Merlin Communication Protocols.

Some Perspectives on Computational and Communication Complexity

Lecture notes on descriptional complexity and randomness

Structure in Communication Complexity and Constant-Cost Complexity Classes

Unbounded quantum advantage in communication complexity measured by distinguishability

The Communication Complexity of Optimization

Improving the Bit Complexity of Communication for Distributed Convex Optimization

On Arthur Merlin Games in Communication Complexity

Stripping syntax from complexity: An information-theoretical perspective on complex systems

The Complexity of Quantum States and Transformations: From Quantum Money to Black Holes

Lower bounds for quantum-inspired classical algorithms via communication complexity

The communication complexity of functions with large outputs

Some Perspectives on Complexity-Based Cryptography

Some perspectives on computational complexity

Communication Lower Bounds for Distributed Convex Optimization: Partition Data on Features.

Communication Complexity of Graph Isomorphism, Coloring, and Distance Games

Separation between Classical and Quantum Winning Strategies for the Matching Game

Programming with Quantum Communication

On the Hardness of Dominant Strategy Mechanism Design