Abstract:This paper studies the Binary Neural Networks (BNNs) in which weights and activations are both binarized into 1-bit values, thus greatly reducing the memory usage and computational complexity. Since the modern deep neural networks are of sophisticated design with complex architecture for the accuracy reason, the diversity on distributions of weights and activations is very high. Therefore, the conventional sign function cannot be well used for effectively binarizing full-precision values in BNNs. To this end, we present a simple yet effective approach called AdaBin to adaptively obtain the optimal binary sets $\{b_1, b_2\}$ ($b_1, b_2\in \mathbb{R}$) of weights and activations for each layer instead of a fixed set (\textit{i.e.}, $\{-1, +1\}$). In this way, the proposed method can better fit different distributions and increase the representation ability of binarized features. In practice, we use the center position and distance of 1-bit values to define a new binary quantization function. For the weights, we propose an equalization method to align the symmetrical center of binary distribution to real-valued distribution, and minimize the Kullback-Leibler divergence of them. Meanwhile, we introduce a gradient-based optimization method to get these two parameters for activations, which are jointly trained in an end-to-end manner. Experimental results on benchmark models and datasets demonstrate that the proposed AdaBin is able to achieve state-of-the-art performance. For instance, we obtain a 66.4% Top-1 accuracy on the ImageNet using ResNet-18 architecture, and a 69.4 mAP on PASCAL VOC using SSD300. The PyTorch code is available at \url{https://github.com/huawei-noah/Efficient-Computing/tree/master/BinaryNetworks/AdaBin} and the MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/AdaBin}.

Learning Adaptive Gradients for Binary Neural Networks

Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks

Gradient Corrected Approximation for Binary Neural Networks.

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

SGDAT: an Optimization Method for Binary Neural Networks

Fast and Slow Gradient Approximation for Binary Neural Network Optimization

Gradient Matters: Designing Binarized Neural Networks Via Enhanced Information-Flow

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

Adaptive Linear Unit for Accurate Binary Neural Networks

Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training

Blended coarse gradient descent for full quantization of deep neural networks

SA-BNN: State-Aware Binary Neural Network.

MetaGrad: Adaptive Gradient Quantization with Hypernetworks

Learning to Binarize Convolutional Neural Networks with Adaptive Neural Encoder

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

Binary Convolutional Neural Network with High Accuracy and Compression Rate

Training Multi-Layer Binary Neural Networks With Local Binary Error Signals

A Gradient Optimization and Manifold Preserving based Binary Neural Network for Point Cloud

Binarizing by Classification: is Soft Function Really Necessary?