Abstract:This paper studies the Binary Neural Networks (BNNs) in which weights and activations are both binarized into 1-bit values, thus greatly reducing the memory usage and computational complexity. Since the modern deep neural networks are of sophisticated design with complex architecture for the accuracy reason, the diversity on distributions of weights and activations is very high. Therefore, the conventional sign function cannot be well used for effectively binarizing full-precision values in BNNs. To this end, we present a simple yet effective approach called AdaBin to adaptively obtain the optimal binary sets $\{b_1, b_2\}$ ($b_1, b_2\in \mathbb{R}$) of weights and activations for each layer instead of a fixed set (\textit{i.e.}, $\{-1, +1\}$). In this way, the proposed method can better fit different distributions and increase the representation ability of binarized features. In practice, we use the center position and distance of 1-bit values to define a new binary quantization function. For the weights, we propose an equalization method to align the symmetrical center of binary distribution to real-valued distribution, and minimize the Kullback-Leibler divergence of them. Meanwhile, we introduce a gradient-based optimization method to get these two parameters for activations, which are jointly trained in an end-to-end manner. Experimental results on benchmark models and datasets demonstrate that the proposed AdaBin is able to achieve state-of-the-art performance. For instance, we obtain a 66.4% Top-1 accuracy on the ImageNet using ResNet-18 architecture, and a 69.4 mAP on PASCAL VOC using SSD300. The PyTorch code is available at \url{https://github.com/huawei-noah/Efficient-Computing/tree/master/BinaryNetworks/AdaBin} and the MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/AdaBin}.

Improved Memristive Binarized Neural Networks Using Transformer_DCBNN Architecture with CBAM Attention Mechanism

Memristive KDG-BNN: Memristive binary neural networks trained via knowledge distillation and generative adversarial networks

Efficient Memristive Binary Neural Networks with Spatial Separable Convolutions

Binary Convolutional Neural Network on RRAM.

TB-DNN: A Thin Binarized Deep Neural Network with High Accuracy

PR-CIM: a Variation-Aware Binary-Neural-Network Framework for Process-Resilient Computation-in-memory

Design of High Robustness BNN Inference Accelerator Based on Binary Memristors

A Variation-Aware Binary Neural Network Framework for Process Resilient In-Memory Computations

An Improved RRAM-Based Binarized Neural Network with High Variation-Tolerated Forward/Backward Propagation Module

Memristor-Based Binarized Spiking Neural Networks: Challenges and applications

Tiny but Accurate: A Pruned, Quantized and Optimized Memristor Crossbar Framework for Ultra Efficient DNN Implementation

FBNA: A Fully Binarized Neural Network Accelerator.

Binary Convolutional Neural Network with High Accuracy and Compression Rate

An Improved Hardware Accelaration Architecture of Binary Neural Network With 1T1R Array Based Forward/Backward Propagation Module

BASER: Bit-wise Approximate Compressor Configurable In-SRAM-computing for Energy-Efficient Neural Network Acceleration with Data-aware Weight Remapping Method

An Ultra-Efficient Memristor-Based DNN Framework with Structured Weight Pruning and Quantization Using ADMM

An Energy-Efficient Bagged Binary Neural Network Accelerator

1S-1R Array: Pure-memristor Circuit for Binary Neural Networks

STBNN: Hardware-friendly Spatio-Temporal Binary Neural Network with High Pattern Recognition Accuracy

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

An Energy-Efficient Architecture for Binary Weight Convolutional Neural Networks