Abstract:Recently, due to privacy concerns, distributed machine learning in Wide-Area Networks (DML-WANs) attracts increasing attention and has been widely deployed to promote the widespread application of intelligence services that rely on geographically distributed data. DML-WANs is essentially performing collaboratively federated learning over a combination of servers at both edge and cloud on a large spatial scale. However, efficient model training is challenging for DML-WANs because it is blocked by the high overhead of model parameter synchronization between computing servers over WANs. The reason is that there has a sequential dependency between local model computing and global model synchronization of traditional DML-WANs training methods intrinsically producing a sequential blockage between them, e.g., FedAvg. When the computing heterogeneity and the low WAN bandwidth coexist, a long block of global model synchronization prolongs the training time and leads to low utilization of local computing. Despite many efforts on alleviating synchronization overhead with novel communication technologies and synchronization methods, they still use traditional training patterns with sequential dependency and thereby have very limited improvements, such as FedAsync and ESync. In this article, we propose NBSync, a novel training algorithm for DML-WANs, which greatly speeds up the model training by the parallelism of local computing and global synchronization. NBSync employs a well-designed pipelining scheme, which can properly relax the sequential dependency of local computing and global synchronization and process them in parallel so as to overlap their operating overhead in the time dimension. NBSync also realizes flexible, differentiated and dynamical local computing for workers to maximize the overlap ratio in dynamically heterogeneous training environments. Convergence analysis shows that the convergence rate of NBSync training process is asymptotically equal to that of SSGD, and NBSync has a better convergence efficiency. We implemented the prototype of NBSync based on a popular parameter server system, i.e., MXNET's PS-LITE library, and evaluate its performance on a DML-WANs testbed. Experimental results show that NBSync speeds up training about 1.43×–2.79× than state-of-the-art distributed training algorithms (DTAs) in DML-WANs scenarios where computing heterogeneity and low WAN bandwidth coexist.

HiPS - Hierarchical Parameter Synchronization in Large-Scale Distributed Machine Learning.

Impact of Synchronization Topology on DML Performance: Both Logical Topology and Physical Topology

WBSP: Addressing Stragglers in Distributed Machine Learning with Worker-Busy Synchronous Parallel

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Near-Optimal Topology-adaptive Parameter Synchronization in Distributed DNN Training

Efficient Communication Scheduling for Parameter Synchronization of DML in Data Center Networks

Adaptive Load Balancing for Parameter Servers in Distributed Machine Learning over Heterogeneous Networks

OSP: Boosting Distributed Model Training with 2-Stage Synchronization

Accelerating Distributed Machine Learning by Smart Parameter Server

PSscheduler: A Parameter Synchronization Scheduling Algorithm for Distributed Machine Learning in Reconfigurable Optical Networks

NBSync: Parallelism of Local Computing and Global Synchronization for Fast Distributed Machine Learning in WANs

SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training.

HeterPS: Distributed deep learning with reinforcement learning based scheduling in heterogeneous environments

A scalable and topology configurable protocol for distributed parameter synchronization

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Petrel: Heterogeneity-Aware Distributed Deep Learning Via Hybrid Synchronization

Heterogeneity-Aware Distributed Parameter Servers

BML: A High-performance, Low-cost Gradient Synchronization Algorithm for DML Training

Distributed Machine Learning through Heterogeneous Edge Systems

A Quadratic Synchronization Rule for Distributed Deep Learning