Abstract:The extensive need for computational resources poses a significant obstacle to deploying large-scale Deep Neural Networks (DNN) on devices with constrained resources. At the same time, studies have demonstrated that a significant number of these DNN parameters are redundant and extraneous. In this paper, we introduce a novel approach for learning structured sparse neural networks, aimed at bridging the DNN hardware deployment challenges. We develop a novel regularization technique, termed Weighted Group Sparse Envelope Function (WGSEF), generalizing the Sparse Envelop Function (SEF), to select (or nullify) neuron groups, thereby reducing redundancy and enhancing computational efficiency. The method speeds up inference time and aims to reduce memory demand and power consumption, thanks to its adaptability which lets any hardware specify group definitions, such as filters, channels, filter shapes, layer depths, a single parameter (unstructured), etc. The properties of the WGSEF enable the pre-definition of a desired sparsity level to be achieved at the training convergence. In the case of redundant parameters, this approach maintains negligible network accuracy degradation or can even lead to improvements in accuracy. Our method efficiently computes the WGSEF regularizer and its proximal operator, in a worst-case linear complexity relative to the number of group variables. Employing a proximal-gradient-based optimization technique, to train the model, it tackles the non-convex minimization problem incorporating the neural network loss and the WGSEF. Finally, we experiment and illustrate the efficiency of our proposed method in terms of the compression ratio, accuracy, and inference latency.

Learning Sparse Patterns in Deep Neural Networks

Deep Neural Network Acceleration with Sparse Prediction Layers

Joint Sparsity with Mixed Granularity for Efficient GPU Implementation

Learning Low-Rank Structured Sparsity in Recurrent Neural Networks

SparseConnect: Regularising CNNs on Fully Connected Layers

Learning Structured Sparsity in Deep Neural Networks

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

Exploring the Regularity of Sparse Structure in Convolutional Neural Networks

Improving Deep Neural Network Sparsity Through Decorrelation Regularization

Learning Instance-wise Sparsity for Accelerating Deep Models

A Computing Efficient Hardware Architecture for Sparse Deep Neural Network Computing

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization.

Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularization

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization

Overfitting Remedy by Sparsifying Regularization on Fully-Connected Layers of CNNs.

Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks

Efficient Network Compression Through Smooth-Lasso Constraint

Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning

SparseNet: A Sparse DenseNet for Image Classification

Adaptive Pixel-wise Structured Sparse Network for Efficient CNNs

Exploring the Granularity of Sparsity in Convolutional Neural Networks.