Abstract:This paper works on Binary Neural Networks (BNNs), a promising avenue for efficient deep learning, offering significant reductions in computational overhead and memory footprint to full precision networks. However, the challenge of energy-intensive training and the drop in performance have been persistent issues. Tackling the challenge, prior works focus primarily on task-related inference optimization. Unlike prior works, this study offers an innovative methodology integrating BNNs with cyclic precision training, introducing the CycleBNN. This approach is designed to enhance training efficiency while minimizing the loss in performance. By dynamically adjusting precision in cycles, we achieve a convenient trade-off between training efficiency and model performance. This emphasizes the potential of our method in energy-constrained training scenarios, where data is collected onboard and paves the way for sustainable and efficient deep learning architectures. To gather insights on CycleBNN's efficiency, we conduct experiments on ImageNet, CIFAR-10, and PASCAL-VOC, obtaining competitive performances while using 96.09\% less operations during training on ImageNet, 88.88\% on CIFAR-10 and 96.09\% on PASCAL-VOC. Finally, CycleBNN offers a path towards faster, more accessible training of efficient networks, accelerating the development of practical applications. The PyTorch code is available at \url{<a class="link-external link-https" href="https://github.com/fedeloper/CycleBNN/" rel="external noopener nofollow">this https URL</a>}

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve two major problems encountered in the training process of Binary Neural Networks (BNNs): **the energy - consuming training process** and **performance degradation**. Although Binary Neural Networks have significant computational efficiency and memory - occupancy advantages during the inference stage, their training process still relies on full - precision (32 - bit representation), resulting in low training efficiency and a decline in model performance. #### Specific problems: 1. **Low training efficiency**: The training process of BNNs uses full - precision (32 - bit), which makes the computational overhead in the training process huge, especially in resource - constrained environments (such as edge devices or embedded systems). 2. **Performance degradation**: Although BNNs are more efficient than full - precision networks during inference, during the training process, due to the information loss caused by binarization, the model performance usually declines, especially when dealing with complex tasks. #### Solutions: To solve these problems, the paper proposes **CycleBNN**, a binary neural network method combined with cyclic - precision training. By dynamically adjusting the precision of weights and activations during the training process, CycleBNN can significantly reduce the computational resources required for training while maintaining high model performance. Specifically, CycleBNN achieves a balance between training efficiency and model performance by periodically adjusting the precision (from 2 to 6 bits). This method not only reduces the number of operations in the training process but also improves the generalization ability of the model, and is especially suitable for energy - constrained scenarios. #### Experimental results: - Experiments on datasets such as ImageNet, CIFAR - 10, and PASCAL - VOC show that CycleBNN can reduce the number of training operations by 96.09%, 88.88%, and 96.09% respectively while maintaining a competitive performance level. - This method provides a new way for faster and more efficient network training and accelerates the development of practical applications. In conclusion, CycleBNN solves the problem of low efficiency in the training process of BNNs by introducing cyclic - precision training while maintaining high model performance, and is especially suitable for resource - constrained environments.

CycleBNN: Cyclic Precision Training in Binary Neural Networks

Hyperbolic Binary Neural Network

CPT: Efficient Deep Neural Network Training via Cyclic Precision

Optimizing data-flow in Binary Neural Networks

Resilient Binary Neural Network.

Recurrent Bilinear Optimization for Binary Neural Networks

Enabling On-device Continual Learning with Binary Neural Networks

Ponte: Represent Totally Binary Neural Network Toward Efficiency

Training Multi-Layer Binary Neural Networks With Local Binary Error Signals

"BNN - BN = ?": Training Binary Neural Networks Without Batch Normalization

“BNN - BN = ?”: Training Binary Neural Networks Without Batch Normalization

Better Schedules for Low Precision Training of Deep Neural Networks

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Training binary neural networks without floating point precision

BiPer: Binary Neural Networks using a Periodic Function

Energy-efficient and Robust Cumulative Training with Net2Net Transformation

Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving

Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies.

Binary Neural Networks as a general-propose compute paradigm for on-device computer vision