Abstract:Model binarization is an effective method of compressing neural networks and accelerating their inference process. However, a significant performance gap still exists between the 1-bit model and the 32-bit one. The empirical study shows that binarization causes a great loss of information in the forward and backward propagation. We present a novel Distribution-sensitive Information Retention Network (DIR-Net) that retains the information in the forward and backward propagation by improving internal propagation and introducing external representations. The DIR-Net mainly relies on three technical contributions: (1) Information Maximized Binarization (IMB): minimizing the information loss and the binarization error of weights/activations simultaneously by weight balance and standardization; (2) Distribution-sensitive Two-stage Estimator (DTE): retaining the information of gradients by distribution-sensitive soft approximation by jointly considering the updating capability and accurate gradient; (3) Representation-align Binarization-aware Distillation (RBD): retaining the representation information by distilling the representations between full-precision and binarized networks. The DIR-Net investigates both forward and backward processes of BNNs from the unified information perspective, thereby providing new insight into the mechanism of network binarization. The three techniques in our DIR-Net are versatile and effective and can be applied in various structures to improve BNNs. Comprehensive experiments on the image classification and objective detection tasks show that our DIR-Net consistently outperforms the state-of-the-art binarization approaches under mainstream and compact architectures, such as ResNet, VGG, EfficientNet, DARTS, and MobileNet. Additionally, we conduct our DIR-Net on real-world resource-limited devices which achieves 11.1x storage saving and 5.4x speedup.

Binarized Neural Networks on the ImageNet Classification Task

Sparsity-Inducing Binarized Neural Networks

Hyperbolic Binary Neural Network

On Study of the Binarized Deep Neural Network for Image Classification.

Network Binarization via Contrastive Learning

Improving the Accuracy of Binarized Neural Networks and Application on Remote Sensing Data

Distribution-sensitive Information Retention for Accurate Binary Neural Network

Bi-Real Net V2: Rethinking Non-linearity for 1-Bit CNNs and Going Beyond

Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Self-distribution binary neural networks

SBNN: Slimming binarized neural network

Binarizing Super-Resolution Neural Network Without Batch Normalization.

Training Binary Neural Network Without Batch Normalization for Image Super-Resolution

Efficient Super Resolution Using Binarized Neural Network

An adiabatic method to train binarized artificial neural networks

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

Recurrent Bilinear Optimization for Binary Neural Networks

IE-Net: Information-Enhanced Binary Neural Networks for Accurate Classification

A Simple and General Binarization Method for Image Restoration Neural Networks.

Binarized Neural Architecture Search for Efficient Object Recognition