Asynchronous Stochastic Gradient Descent over Decentralized Datasets

Yubo Du,Keyou You
DOI: https://doi.org/10.1109/tcns.2021.3059848
IF: 4.347
2021-01-01
IEEE Transactions on Control of Network Systems
Abstract:Asynchronous stochastic gradient descent (ASGD) usually works in the centralized setting in which workers retrieve data from a shared training set. This paper focuses on decentralized scenarios where each worker only accesses a subset of the whole training set. We find that due to the heterogeneous properties of the decentralized setting, ASGD will optimize in wrong directions and thus obtain poor solutions. To tackle the issue, a novel algorithm DASGD is proposed for above setting. Our key idea is to form an asymptotically unbiased accurate gradient estimate through reweighting stochastic gradient based on importance sampling technique. Numerical results substantiate the performance of the proposed algorithm in the decentralized setting.
What problem does this paper attempt to address?