Abstract:Neural networks with ReLU activation functions (abbrev. ReLU Networks), have demonstrated their success in many applications. Recently, researchers noticed that ReLU networks are positively scale-invariant (PSI) while the weights are not. This mismatch may lead to undesirable behaviors in the optimization process. Hence, some new algorithms that conduct optimization directly in the \textit{path space} (the path space is proven to be PSI) were developed, such as Stochastic Gradient Descent (SGD) in the path space. However, it is still unknown that whether other deep learning techniques such as batch normalization (BN), could also have their counterparts in the path space. In this paper, we conduct a formal study on the design of BN in the path space. First, we propose \emph{path-reparameterization} of ReLU networks, in which the weights in the networks are reparameterized by path-values. Then, the feedforward and backward propagation of the path-reparameterized networks can calculate the values of the hidden nodes and the gradients in the path space, respectively. Next, we design the a novel way to do batch normalization for the path-reparameterized ReLU networks, called \emph{Path-BN}. Specifically, we notice that, path-reparameterized ReLU NNs have a portion of constant weights which play more critical roles to form the basis of the path space. We propose to exclude these constant weights when doing batch normalization and prove that, by doing so, the scale and the direction of the trained parameters can be more effectively decoupled during training. Finally, we conduct experiments on benchmark datasets. The results show that our proposed Path-BN can improve the performance of the optimization algorithms in the path space.

BNReLU: Combine Batch Normalization and Rectified Linear Unit to Reduce Hardware Overhead.

Improving Fault Tolerance for Reliable DNN Using Boundary-Aware Activation

A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network

Batch-Normalization-based Soft Filter Pruning for Deep Convolutional Neural Networks

EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks.

L1 -Norm Batch Normalization for Efficient Training of Deep Neural Networks

Restructuring Batch Normalization to Accelerate CNN Training

FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

Symmetric-threshold ReLU for Fast and Nearly Lossless ANN-SNN Conversion

Weight initialization based‐rectified linear unit activation function to improve the performance of a convolutional neural network model

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Generalized Batch Normalization: Towards Accelerating Deep Neural Networks

RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs

Path-BN - Towards effective batch normalization in the Path Space for ReLU networks.

Competition-based Adaptive ReLU for Deep Neural Networks

ReActXGB: A Hybrid Binary Convolutional Neural Network Architecture for Improved Performance and Computational Efficiency

Decorrelated Batch Normalization

Efficient Integer-Arithmetic-Only Convolutional Networks with Bounded ReLU

Efficient Integer-Arithmetic-Only Convolutional Neural Networks

First Realization of Batch Normalization in Flash-Based Binary Neural Networks Using a Single Voltage Shifter

Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance