Abstract:This paper studies the Binary Neural Networks (BNNs) in which weights and activations are both binarized into 1-bit values, thus greatly reducing the memory usage and computational complexity. Since the modern deep neural networks are of sophisticated design with complex architecture for the accuracy reason, the diversity on distributions of weights and activations is very high. Therefore, the conventional sign function cannot be well used for effectively binarizing full-precision values in BNNs. To this end, we present a simple yet effective approach called AdaBin to adaptively obtain the optimal binary sets $\{b_1, b_2\}$ ($b_1, b_2\in \mathbb{R}$) of weights and activations for each layer instead of a fixed set (\textit{i.e.}, $\{-1, +1\}$). In this way, the proposed method can better fit different distributions and increase the representation ability of binarized features. In practice, we use the center position and distance of 1-bit values to define a new binary quantization function. For the weights, we propose an equalization method to align the symmetrical center of binary distribution to real-valued distribution, and minimize the Kullback-Leibler divergence of them. Meanwhile, we introduce a gradient-based optimization method to get these two parameters for activations, which are jointly trained in an end-to-end manner. Experimental results on benchmark models and datasets demonstrate that the proposed AdaBin is able to achieve state-of-the-art performance. For instance, we obtain a 66.4% Top-1 accuracy on the ImageNet using ResNet-18 architecture, and a 69.4 mAP on PASCAL VOC using SSD300. The PyTorch code is available at \url{https://github.com/huawei-noah/Efficient-Computing/tree/master/BinaryNetworks/AdaBin} and the MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/AdaBin}.

Rectified Linear Units Improve Restricted Boltzmann Machines

Training Restricted Boltzmann Machines with Binary Synapses Using the Bayesian Learning Rule

Shallow Binary Features Enhance the Robustness of Deep Convolutional Neural Networks

Restricted Boltzmann Machine with Adaptive Local Hidden Units.

Adjustable Bounded Rectifiers: Towards Deep Binary Representations

Restricted Boltzmann Machines: Introduction and Review

ReLUs Are Sufficient for Learning Implicit Neural Representations

Improving neural networks with bunches of neurons modeled by Kumaraswamy units: Preliminary study

Rectified Binary Convolutional Networks with Generative Adversarial Learning

Hyperbolic Linear Units For Deep Convolutional Neural Networks

An Overview of Restricted Boltzmann Machines

Rectified Binary Network for Single-Image Super-Resolution

Linear-Time Sequence Classification using Restricted Boltzmann Machines

Sequence Classification Restricted Boltzmann Machines With Gated Units

Monotone deep Boltzmann machines

Recurrent Bilinear Optimization for Binary Neural Networks

Training Multi-Layer Binary Neural Networks With Local Binary Error Signals

Network Binarization via Contrastive Learning

Binarized Weight Error Networks With a Transition Regularization Term

A Cyclic Contrastive Divergence Learning Algorithm for High-Order RBMs

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets