Abstract:Binary Neural Networks~(BNNs) have been proven to be highly effective for deploying deep neural networks on mobile and embedded platforms. Most existing works focus on minimizing quantization errors, improving representation ability, or designing gradient approximations to alleviate gradient mismatch in BNNs, while leaving the weight sign flipping, a critical factor for achieving powerful BNNs, untouched. In this paper, we investigate the efficiency of weight sign updates in BNNs. We observe that, for vanilla BNNs, over 50\% of the weights remain their signs unchanged during training, and these weights are not only distributed at the tails of the weight distribution but also universally present in the vicinity of zero. We refer to these weights as ``silent weights'', which slow down convergence and lead to a significant accuracy degradation. Theoretically, we reveal this is due to the independence of the BNNs gradient from the latent weight distribution. To address the issue, we propose Overcome Silent Weights~(OvSW). OvSW first employs Adaptive Gradient Scaling~(AGS) to establish a relationship between the gradient and the latent weight distribution, thereby improving the overall efficiency of weight sign updates. Additionally, we design Silence Awareness Decaying~(SAD) to automatically identify ``silent weights'' by tracking weight flipping state, and apply an additional penalty to ``silent weights'' to facilitate their flipping. By efficiently updating weight signs, our method achieves faster convergence and state-of-the-art performance on CIFAR10 and ImageNet1K dataset with various architectures. For example, OvSW obtains 61.6\% and 65.5\% top-1 accuracy on the ImageNet1K using binarized ResNet18 and ResNet34 architecture respectively. Codes are available at \url{<a class="link-external link-https" href="https://github.com/JingyangXiang/OvSW" rel="external noopener nofollow">this https URL</a>}.

Applying BinaryWeights in Neural Networks for Sequential Text Recognition

Deep Spiking Neural Networks with Binary Weights for Object Recognition

OvSW: Overcoming Silent Weights for Accurate Binary Neural Networks

Binary Neural Networks for Speech Recognition

Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization

STBNN: Hardware-friendly Spatio-Temporal Binary Neural Network with High Pattern Recognition Accuracy

An adiabatic method to train binarized artificial neural networks

Document Image Binarization with Feedback for Improving Character Segmentation

Bi-Real Net V2: Rethinking Non-linearity for 1-Bit CNNs and Going Beyond

An Energy and Memory Efficient Speaker Verification System Based on Binary Neural Networks

Braille Recognition Using Deep Learning

Boosting Binary Neural Networks via Dynamic Thresholds Learning

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance.

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

TP-ADMM: An Efficient Two-Stage Framework for Training Binary Neural Networks

BiFSMN: Binary Neural Network for Keyword Spotting

Gradient Matters: Designing Binarized Neural Networks Via Enhanced Information-Flow

On Study of the Binarized Deep Neural Network for Image Classification.

TB-DNN: A Thin Binarized Deep Neural Network with High Accuracy

Low-complex and Highly-performed Binary Residual Neural Network for Small-footprint Keyword Spotting