Abstract:Neural networks with ReLU activation functions (abbrev. ReLU Networks), have demonstrated their success in many applications. Recently, researchers noticed that ReLU networks are positively scale-invariant (PSI) while the weights are not. This mismatch may lead to undesirable behaviors in the optimization process. Hence, some new algorithms that conduct optimization directly in the \textit{path space} (the path space is proven to be PSI) were developed, such as Stochastic Gradient Descent (SGD) in the path space. However, it is still unknown that whether other deep learning techniques such as batch normalization (BN), could also have their counterparts in the path space. In this paper, we conduct a formal study on the design of BN in the path space. First, we propose \emph{path-reparameterization} of ReLU networks, in which the weights in the networks are reparameterized by path-values. Then, the feedforward and backward propagation of the path-reparameterized networks can calculate the values of the hidden nodes and the gradients in the path space, respectively. Next, we design the a novel way to do batch normalization for the path-reparameterized ReLU networks, called \emph{Path-BN}. Specifically, we notice that, path-reparameterized ReLU NNs have a portion of constant weights which play more critical roles to form the basis of the path space. We propose to exclude these constant weights when doing batch normalization and prove that, by doing so, the scale and the direction of the trained parameters can be more effectively decoupled during training. Finally, we conduct experiments on benchmark datasets. The results show that our proposed Path-BN can improve the performance of the optimization algorithms in the path space.

Path-BN: Towards Effective Batch Normalization in the Path Space for ReLU Networks Supplementary Materials

Path-BN - Towards effective batch normalization in the Path Space for ReLU networks.

Path-BN: Towards effective batch normalization in the Path Space for ReLU networks

Path Space for Recurrent Neural Networks with ReLU Activations

Capacity control of ReLU neural networks by basis-path norm

On Centralization and Unitization of Batch Normalization for Deep ReLU Neural Networks

Path-metrics, pruning, and generalization

Functional vs. parametric equivalence of ReLU networks

Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks

Overparameterized ReLU Neural Networks Learn the Simplest Model: Neural Isometry and Phase Transitions

Numerical influence of ReLU'(0) on backpropagation

Generalized Batch Normalization: Towards Accelerating Deep Neural Networks

-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

Phase Diagram for Two-layer ReLU Neural Networks at Infinite-width Limit.

Analysis on Gradient Propagation in Batch Normalized Residual Networks

Interpreting the basis path set in neural networks

The effect of Leaky ReLUs on the training and generalization of overparameterized networks

Neural networks with ReLU powers need less depth