Abstract:Although deep neural networks (DNNs) are being a revolutionary power to open up the AI era, the notoriously huge hardware overhead has challenged their applications. Recently, several binary and ternary networks, in which the costly multiply-accumulate operations can be replaced by accumulations or even binary logic operations, make the on-chip training of DNNs quite promising. Therefore there is a pressing need to build an architecture that could subsume these networks under a unified framework that achieves both higher performance and less overhead. To this end, two fundamental issues are yet to be addressed. The first one is how to implement the back propagation when neuronal activations are discrete. The second one is how to remove the full-precision hidden weights in the training phase to break the bottlenecks of memory/computation consumption. To address the first issue, we present a multi-step neuronal activation discretization method and a derivative approximation technique that enable the implementing the back propagation algorithm on discrete DNNs. While for the second issue, we propose a discrete state transition (DST) methodology to constrain the weights in a discrete space without saving the hidden weights. Through this way, we build a unified framework that subsumes the binary or ternary networks as its special cases, and under which a heuristic algorithm is provided at the website https://github.com/AcrossV/Gated-XNOR. More particularly, we find that when both the weights and activations become ternary values, the DNNs can be reduced to sparse binary networks, termed as gated XNOR networks (GXNOR-Nets) since only the event of non-zero weight and non-zero activation enables the control gate to start the XNOR logic operations in the original binary networks. This promises the event-driven hardware design for efficient mobile intelligence. We achieve advanced performance compared with state-of-the-art algorithms. Furthermore, the computational sparsity and the number of states in the discrete space can be flexibly modified to make it suitable for various hardware platforms.

Weights and Structure Determination of Pruning-While-Growing Type for 3-Input Power-Activation Feed-Forward Neuronet

Class-Aware Pruning for Efficient Neural Networks

Efficient Structure Slimming for Spiking Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

A structure optimisation algorithm for feedforward neural network construction

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Weight-Dependent Gates for Network Pruning

Pruning of Deep Spiking Neural Networks Through Gradient Rewiring.

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Differentiable channel pruning guided via attention mechanism: a novel neural network pruning approach

Gated XNOR Networks: Deep Neural Networks with Ternary Weights and Activations under a Unified Discretization Framework.

TWD-SFNN: Three-way decisions with a single hidden layer feedforward neural network

State Transition of Dendritic Spines Improves Learning of Sparse Spiking Neural Networks.

A Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers

SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency

Adaptive Search-and-Training for Robust and Efficient Network Pruning

Structured Pruning of Recurrent Neural Networks through Neuron Selection

GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework

Online training and pruning of photonic neural networks

Global probability distribution structure-sparsity filter pruning for edge fault diagnosis in resource constrained wireless sensor networks

A systematic DNN weight pruning framework based on symmetric accelerated stochastic ADMM