Abstract:Our work addresses the well-known open problem of distributed computing of bilinear functions of two correlated sources ${\bf A}$ and ${\bf B}$. In a setting with two nodes, with the first node having access to ${\bf A}$ and the second to ${\bf B}$, we establish bounds on the optimal sum-rate that allows a receiver to compute an important class of non-linear functions, and in particular bilinear functions, including dot products $\langle {\bf A},{\bf B}\rangle$, and general matrix products ${\bf A}^{\intercal}{\bf B}$ over finite fields. The bounds are tight, for large field sizes, for which case we can derive the exact fundamental performance limits for all problem dimensions and a large class of sources. Our achievability scheme involves the design of non-linear transformations of ${\bf A}$ and ${\bf B}$, which are carefully calibrated to work synergistically with the structured linear encoding scheme by Körner and Marton. The subsequent converse derived here, calibrates the Han-Kobayashi approach to yield a relatively tight converse on the sum rate. We also demonstrate unbounded compression gains over Slepian-Wolf coding, depending on the source correlations. In the end, our work derives fundamental limits for distributed computing of a crucial class of functions, succinctly capturing the computation structures and source correlations. Our findings are subsequently applied to the practical master-workers-receiver framework, where each of $N$ distributed workers has a bounded memory reflecting a bounded computational capability. By combining the above scheme with the polynomial code framework, we design novel structured polynomial codes for distributed matrix multiplication, and show that our codes can surpass the performance of the existing state of art, while also adapting these new codes to support chain matrix multiplications and information-theoretically secure computations.

Communication Lower Bounds and Optimal Algorithms for Symmetric Matrix Computations

Communication Lower Bounds and Optimal Algorithms for Multiple Tensor-Times-Matrix Computation

Communication-Optimal Parallel Algorithm for Strassen's Matrix Multiplication

Avoiding communication in sparse matrix computations

A Communication-Avoiding Parallel Algorithm for the Symmetric Eigenvalue Problem

Communication Avoiding Block Low-Rank Parallel Multifrontal Triangular Solve with Many Right-Hand Sides

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal Matrix Factorizations

The Communication Complexity of Approximating Matrix Rank

Skew-Symmetric Matrix Decompositions on Shared-Memory Architectures

Graph Expansion Analysis for Communication Costs of Fast Rectangular Matrix Multiplication

A Communication Avoiding and Reducing Algorithm for Symmetric Eigenproblem for Very Small Matrices

COSTA: Communication-Optimal Shuffle and Transpose Algorithm with Process Relabeling

Collective Communication Optimization for Solving Linear Algebraic Equations

Minimizing Computation and Communication Costs of Two-Sided Secure Distributed Matrix Multiplication under Arbitrary Collusion Pattern

On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal LU Factorization

The Improved Krylov Subspace Methods for Large and Sparse Linear Systems on Bulk Synchronous Parallel Architectures

Structured Codes for Distributed Matrix Multiplication

Research on the Secure Multi-Party Computation of Some Linear Algebra Problems

Low-Bandwidth Matrix Multiplication: Faster Algorithms and More General Forms of Sparsity

Matrix Multiplication and Number On the Forehead Communication

Arrow Matrix Decomposition: A Novel Approach for Communication-Efficient Sparse Matrix Multiplication