Abstract:Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive studies of gradient compression for data parallel-style training, compressing the activations for models trained with pipeline parallelism is still an open problem. In this paper, we propose AC-SGD, a novel activation compression algorithm for communication-efficient pipeline parallelism training over slow networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AC-SGD compresses the changes of the activations. This allows us to show, to the best of our knowledge for the first time, that one can still achieve $O(1/\sqrt{T})$ convergence rate for non-convex objectives under activation compression, without making assumptions on gradient unbiasedness that do not hold for deep learning models with non-linear activation <a class="link-external link-http" href="http://functions.We" rel="external noopener nofollow">this http URL</a> then show that AC-SGD can be optimized and implemented efficiently, without additional end-to-end runtime <a class="link-external link-http" href="http://overhead.We" rel="external noopener nofollow">this http URL</a> evaluated AC-SGD to fine-tune language models with up to 1.5 billion parameters, compressing activations to 2-4 <a class="link-external link-http" href="http://bits.AC" rel="external noopener nofollow">this http URL</a>-SGD provides up to 4.3X end-to-end speed-up in slower networks, without sacrificing model quality. Moreover, we also show that AC-SGD can be combined with state-of-the-art gradient compression algorithms to enable "end-to-end communication compression: All communications between machines, including model gradients, forward activations, and backward gradients are compressed into lower <a class="link-external link-http" href="http://precision.This" rel="external noopener nofollow">this http URL</a> provides up to 4.9X end-to-end speed-up, without sacrificing model quality.

Long short-term memory with activation on gradient

Activated Gradients for Deep Neural Networks

Normalized Activation Function: Toward Better Convergence

Accelerating CNN Training by Sparsifying Activation Gradients

Efficient Search for Customized Activation Functions with Gradient Descent

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Activation function optimization method: Learnable series linear units (LSLUs)

ELSTM: An improved long short‐term memory network language model for sequence learning

Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning

Activate or Not: Learning Customized Activation

Stochastic Gradient Descent with Variance Reduction Technique

Regularized Flexible Activation Function Combinations for Deep Neural Networks

Learning Longer Memory in Recurrent Neural Networks

LLR: Learning Learning Rates by LSTM for Training Neural Networks.

Fast gradient-free activation maximization for neurons in spiking neural networks

Deterministic convergence analysis for regularized long short-term memory and its application to regression and multi-classification problems

Adding gradient noise improves learning for very deep networks

Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees

Adaptive-saturated RNN: Remember more with less instability

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization