Abstract:Model binarization is an effective method of compressing neural networks and accelerating their inference process. However, a significant performance gap still exists between the 1-bit model and the 32-bit one. The empirical study shows that binarization causes a great loss of information in the forward and backward propagation. We present a novel Distribution-sensitive Information Retention Network (DIR-Net) that retains the information in the forward and backward propagation by improving internal propagation and introducing external representations. The DIR-Net mainly relies on three technical contributions: (1) Information Maximized Binarization (IMB): minimizing the information loss and the binarization error of weights/activations simultaneously by weight balance and standardization; (2) Distribution-sensitive Two-stage Estimator (DTE): retaining the information of gradients by distribution-sensitive soft approximation by jointly considering the updating capability and accurate gradient; (3) Representation-align Binarization-aware Distillation (RBD): retaining the representation information by distilling the representations between full-precision and binarized networks. The DIR-Net investigates both forward and backward processes of BNNs from the unified information perspective, thereby providing new insight into the mechanism of network binarization. The three techniques in our DIR-Net are versatile and effective and can be applied in various structures to improve BNNs. Comprehensive experiments on the image classification and objective detection tasks show that our DIR-Net consistently outperforms the state-of-the-art binarization approaches under mainstream and compact architectures, such as ResNet, VGG, EfficientNet, DARTS, and MobileNet. Additionally, we conduct our DIR-Net on real-world resource-limited devices which achieves 11.1x storage saving and 5.4x speedup.

Learning to Binarize Convolutional Neural Networks with Adaptive Neural Encoder

Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks

Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks

Network Binarization via Contrastive Learning

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

Binary Convolutional Neural Network with High Accuracy and Compression Rate

Self-distribution binary neural networks

Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization

Model encoding of binary neural networks

Binarized Low-light Raw Video Enhancement

CBin-NN: An Inference Engine for Binarized Neural Networks

A High-Efficiency FPGA-Based Accelerator for Binarized Neural Network

SBNN: Slimming binarized neural network

Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers

Build a Compact Binary Neural Network Through Bit-level Sensitivity and Data Pruning.

Gradient Corrected Approximation for Binary Neural Networks.

Distribution-sensitive Information Retention for Accurate Binary Neural Network

FBNA: A Fully Binarized Neural Network Accelerator.

Bi-Real Net V2: Rethinking Non-linearity for 1-Bit CNNs and Going Beyond

Rotated Binary Neural Network

FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond