Abstract:In terms of memory footprint requirement and computing speed, the binary neural networks (BNNs) have great advantages in power-aware deployment applications, such as AIoT edge terminals, wearable and portable devices, etc. However, the networks’ binarization process inevitably brings considerable information losses, and further leads to accuracy deterioration. To tackle these problems, we initiate analyzing from a perspective of the information theory, and manage to improve the networks information capacity. Based on the analyses, our work has two primary contributions: the first is a newly proposed median loss (ML) regularization technique. It improves the binary weights distribution more evenly, and consequently increases the information capacity of BNNs greatly. The second is the batch median of activations (BMA) method. It raises the entropy of activations by subtracting a median value, and simultaneously lowers the quantization error by computing separate scaling factors for the positive and negative activations procedure. Experiment results prove that the proposed methods utilized in ResNet-18 and ResNet-34 individually outperform the Bi-Real baseline by 1.3% and 0.9% Top-1 accuracy on the ImageNet 2012. Proposed ML and BMA for the storage cost and calculation complexity increments are minor and negligible. Additionally, comprehensive experiments also prove that our methods can be applicable and embedded into the present popular BNN networks with accuracy improvement and negligible overhead increment.

Feature Weight Tuning for Recursive Neural Networks.

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

WeNet: Weighted Networks for Recurrent Network Architecture Search

ReCU - Reviving the Dead Weights in Binary Neural Networks.

Structured Pruning of Recurrent Neural Networks through Neuron Selection

Importance estimate of features via analysis of their weight and gradient profile

Residual Connections Harm Generative Representation Learning

Increasing Information Entropy of Both Weights and Activations for the Binary Neural Networks

Randomly Initialized Subnetworks with Iterative Weight Recycling

Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Recursive Binary Neural Network Learning Model with 2.28b/Weight Storage Requirement

Differential Evolution Based Layer-Wise Weight Pruning for Compressing Deep Neural Networks

Weightless: Lossy Weight Encoding For Deep Neural Network Compression

Learning to Weight Samples for Dynamic Early-Exiting Networks.

WeightNet: Revisiting the Design Space of Weight Networks

Learning Useful Representations of Recurrent Neural Network Weight Matrices

Adaptive Multi-Compositionality for Recursive Neural Network Models

Learning to Auto Weight: Entirely Data-Driven and Highly Efficient Weighting Framework

Reweighted Alternating Direction Method of Multipliers for DNN weight pruning

Enhancing the Regularization Effect of Weight Pruning in Artificial Neural Networks