vqSGD: Vector Quantized Stochastic Gradient Descent
Venkata Gandikota,Daniel Kane,Raj Kumar Maity,Arya Mazumdar
DOI: https://doi.org/10.1109/tit.2022.3161620
IF: 2.5
2022-01-01
IEEE Transactions on Information Theory
Abstract:In this work, we present a family of vector quantization schemes vqSGD (Vector-Quantized Stochastic Gradient Descent) that provide an asymptotic reduction in the communication cost with convergence guarantees in first-order distributed optimization. In the process we derive the following fundamental information theoretic fact: bits are necessary and sufficient (up to an additive term) to describe an unbiased estimator for any in the -dimensional unit sphere, under the constraint that almost surely, . In particular, we consider a randomized scheme based on the convex hull of a point set, that returns an unbiased estimator of a -dimensional gradient vector with almost surely bounded norm. We provide multiple efficient instances of our scheme, that are near optimal, and require bits of communication at the expense of tolerable increase in error. The instances of our quantization scheme are obtained using well-known families of binary error-correcting codes and provide a smooth tradeoff between the communication and the estimation error of quantization. Furthermore, we show that vqSGD also off-rs automatic privacy guarantees.
computer science, information systems,engineering, electrical & electronic