Abstract:Recently, due to privacy concerns, distributed machine learning in Wide-Area Networks (DML-WANs) attracts increasing attention and has been widely deployed to promote the widespread application of intelligence services that rely on geographically distributed data. DML-WANs is essentially performing collaboratively federated learning over a combination of servers at both edge and cloud on a large spatial scale. However, efficient model training is challenging for DML-WANs because it is blocked by the high overhead of model parameter synchronization between computing servers over WANs. The reason is that there has a sequential dependency between local model computing and global model synchronization of traditional DML-WANs training methods intrinsically producing a sequential blockage between them, e.g., FedAvg. When the computing heterogeneity and the low WAN bandwidth coexist, a long block of global model synchronization prolongs the training time and leads to low utilization of local computing. Despite many efforts on alleviating synchronization overhead with novel communication technologies and synchronization methods, they still use traditional training patterns with sequential dependency and thereby have very limited improvements, such as FedAsync and ESync. In this article, we propose NBSync, a novel training algorithm for DML-WANs, which greatly speeds up the model training by the parallelism of local computing and global synchronization. NBSync employs a well-designed pipelining scheme, which can properly relax the sequential dependency of local computing and global synchronization and process them in parallel so as to overlap their operating overhead in the time dimension. NBSync also realizes flexible, differentiated and dynamical local computing for workers to maximize the overlap ratio in dynamically heterogeneous training environments. Convergence analysis shows that the convergence rate of NBSync training process is asymptotically equal to that of SSGD, and NBSync has a better convergence efficiency. We implemented the prototype of NBSync based on a popular parameter server system, i.e., MXNET's PS-LITE library, and evaluate its performance on a DML-WANs testbed. Experimental results show that NBSync speeds up training about 1.43×–2.79× than state-of-the-art distributed training algorithms (DTAs) in DML-WANs scenarios where computing heterogeneity and low WAN bandwidth coexist.

DSANA: A Distributed Machine Learning Acceleration Solution Based on Dynamic Scheduling and Network Acceleration

Efficient Communication Scheduling for Parameter Synchronization of DML in Data Center Networks

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

Accelerating Distributed Learning in Non-Dedicated Environments

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

Slim-DP: A Multi-Agent System for Communication-Efficient Distributed Deep Learning

An Adaptive Synchronous Parallel Strategy for Distributed Machine Learning

Adaptive Load Balancing for Parameter Servers in Distributed Machine Learning over Heterogeneous Networks

No Worker Left (Too Far) Behind: Dynamic Hybrid Synchronization for In‐Network ML Aggregation

Traffic Management for Distributed Machine Learning in RDMA-enabled Data Center Networks.

Straggler-Aware Gradient Aggregation for Large-Scale Distributed Deep Learning System

Accelerating Distributed Machine Learning by Smart Parameter Server

NBSync: Parallelism of Local Computing and Global Synchronization for Fast Distributed Machine Learning in WANs

An Efficient Distributed Machine Learning Framework in Wireless D2D Networks: Convergence Analysis and System Implementation

When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning

DBS: Dynamic Batch Size For Distributed Deep Neural Network Training

A Load-Balancing Strategy Based on Multi-Task Learning in a Distributed Training Environment

RDMA Based Congestion Control Strategy for DML Training Optimization in Data Center Networks

GSSP: Eliminating Stragglers Through Grouping Synchronous for Distributed Deep Learning in Heterogeneous Cluster.