Abstract:In recent years, deep learning models have been successfully applied to large-scale data analysis, including image classification, video caption, natural language processing, etc. Large-scale data analyses take advantage of parallel computing to accelerate the speed of model training, in which data parallelism has become the dominant method for deep learning model training due to its high throughput rate. Synchronous stochastic gradient descent optimization becomes a well-recognized optimization method to ensure model convergence, but the overhead of gradients synchronization increases linearly as the number of workers increases, causing a huge waste of time. Although some efficiency-first asynchronous methods have been proposed, these methods cannot guarantee their convergence in large-scale distributed training. To solve this problem, we propose an efficient pseudo-synchronous approach that updates the network with the previous gradient, performing the synchronization of a new gradient to overlap computation and synchronization. This idea will obviously affect the normal convergence of the model, so we propose a novel adaptive exponential smoothing predicted gradient algorithm for model optimization, which can adaptively adjust the confidence coefficient of the history gradient to ensure the normal convergence of the training process. Experiments prove that our method can speed up the training process and achieve a comparable accuracy rate with standard synchronous SGD. Besides, our method has more efficient weak scalability compared to the traditional synchronous SGD and those in previous related work. We apply our methods to image recognition and video caption applications at most 12288 cores with strong scalability on Tianhe II. Evaluations show that, when configured appropriately, our method attains near-linear scalability using 128 nodes. We get 93.4% weak scaling efficiency on 64 nodes, 90.5% on 128 nodes.

A Fast Distributed Accelerated Gradient Algorithm for Big Data Classification.

Fast Classification Algorithms Via Distributed Accelerated Alternating Direction Method of Multipliers.

Accelerated Stochastic ADMM with Variance Reduction

Distributed Stochastic Alternating Direction Method of Multipliers for Big Data Classification

Distributed Classification for Imbalanced Big Data in Distributed Environments

A Fast Distributed Classification Algorithm for Large-Scale Imbalanced Data

Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification

A Fast Alternating Direction Method of Multipliers Algorithm for Big Data Applications.

Variance Reduced Diffusion Adaptation for Online Learning over Networks

Accelerated Stochastic Gradient Method for Support Vector Machines Classification with Additive Kernel

Cost-Sensitive Alternating Direction Method Of Multipliers For Large-Scale Classification

Accelerated Variance Reduction Stochastic ADMM for Large-Scale Machine Learning

Peering Beyond the Gradient Veil with Distributed Auto Differentiation

An Accelerated Distributed Gradient Method with Local Memory

Accelerating Massively Distributed Deep Learning Through Efficient Pseudo-Synchronous Update Method

A Novel Stochastic Gradient Descent Algorithm Based on Grouping over Heterogeneous Cluster Systems for Distributed Deep Learning

Decoupled Asynchronous Proximal Stochastic Gradient Descent with Variance Reduction.

CADA: Communication-Adaptive Distributed Adam

Feature-Distributed SVRG for High-Dimensional Linear Classification

Distributed Adaptive Subgradient Algorithms for Online Learning over Time-Varying Networks

Accelerating SGD Using Flexible Variance Reduction on Large-Scale Datasets