Abstract:Recently, deep learning has made remarkable strides, especially with generative modeling, such as large language models and probabilistic diffusion models. However, training these models often involves significant computational resources, requiring billions of petaFLOPs. This high resource consumption results in substantial energy usage and a large carbon footprint, raising critical environmental concerns. Back-propagation (BP) is a major source of computational expense during training deep learning models. To advance research on energy-efficient training and allow for sparse learning on any machine and device, we propose a general, energy-efficient convolution module that can be seamlessly integrated into any deep learning architecture. Specifically, we introduce channel-wise sparsity with additional gradient selection schedulers during backward based on the assumption that BP is often dense and inefficient, which can lead to over-fitting and high computational consumption. Our experiments demonstrate that our approach reduces 40\% computations while potentially improving model performance, validated on image classification and generation tasks. This reduction can lead to significant energy savings and a lower carbon footprint during the research and development phases of large-scale AI systems. Additionally, our method mitigates over-fitting in a manner distinct from Dropout, allowing it to be combined with Dropout to further enhance model performance and reduce computational resource usage. Extensive experiments validate that our method generalizes to a variety of datasets and tasks and is compatible with a wide range of deep learning architectures and modules. Code is publicly available at <a class="link-external link-https" href="https://github.com/lujiazho/ssProp" rel="external noopener nofollow">this https URL</a>.

Efficient Neural Network Training Via Forward and Backward Propagation Sparsification

Efficient neural network training via forward and backward propagation sparsification

Deep Neural Network Acceleration with Sparse Prediction Layers

Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training.

Memorized Sparse Backpropagation.

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Training Simplification and Model Simplification for Deep Learning : A Minimal Effort Back Propagation Method

SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction

Always-Sparse Training by Growing Connections with Guided Stochastic Exploration

Meprop: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks

A Layer-Based Sparsification Method for Distributed DNN Training.

ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagation

Accelerating CNN Training by Pruning Activation Gradients

Sparse optimization guided pruning for neural networks

Global Sparse Momentum SGD for Pruning Very Deep Neural Networks

Sparse networks from scratch: Faster training without losing performance

Lightweight Spiking Neural Network Training Based on Spike Timing Dependent Backpropagation

Performance of Training Sparse Deep Neural Networks on GPUs

Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning