Abstract:The merits of fast convergence and potentially better performance of the weight normalization family have drawn increasing attention in recent years. These methods use standardization or normalization that changes the weight $\boldsymbol{W}$ to $\boldsymbol{W}'$, which makes $\boldsymbol{W}'$ independent to the magnitude of $\boldsymbol{W}$. Surprisingly, $\boldsymbol{W}$ must be decayed during gradient descent, otherwise we will observe a severe under-fitting problem, which is very counter-intuitive since weight decay is widely known to prevent deep networks from over-fitting. In this paper, we \emph{theoretically} prove that the weight decay term $\frac{1}{2}\lambda||{\boldsymbol{W}}||^2$ merely modulates the effective learning rate for improving objective optimization, and has no influence on generalization when the weight normalization family is compositely employed. Furthermore, we also expose several critical problems when introducing weight decay term to weight normalization family, including the missing of global minimum and training instability. To address these problems, we propose an $\epsilon-$shifted $L_2$ regularizer, which shifts the $L_2$ objective by a positive constant $\epsilon$. Such a simple operation can theoretically guarantee the existence of global minimum, while preventing the network weights from being too small and thus avoiding gradient float overflow. It significantly improves the training stability and can achieve slightly better performance in our practice. The effectiveness of $\epsilon-$shifted $L_2$ regularizer is comprehensively validated on the ImageNet, CIFAR-100, and COCO datasets. Our codes and pretrained models will be released in <a class="link-external link-https" href="https://github.com/implus/PytorchInsight" rel="external noopener nofollow">this https URL</a>.

On objective function, regularizer, and prediction error of a learning algorithm for dealing with multiplicative weight noise

A fault-tolerant regularizer for RBF networks

Multiplicative noise and heavy tails in stochastic optimization

Robust noise-aware algorithm for randomized neural network and its convergence properties

Understanding the Disharmony between Weight Normalization Family and Weight Decay: $ε-$shifted $L_2$ Regularizer

Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations

Empirical strategy for stretching probability distribution in neural-network-based regression

A Hybrid Regularization Approach for Random Vector Functional-Link Networks.

Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

An Analysis of Weight Decay As a Methodology of Reducing Three-Layer Feedforward Artificial Neural Networks for Classification Problems

Convergence and objective functions of noise-injected multilayer perceptrons with hidden multipliers

On the weight dynamics of learning networks

Neural Network for a Class of Sparse Optimization with L0-regularization.

Huber Loss Based Distributed Robust Learning Algorithm for Random Vector Functional-Link Network

On the Robustness of Noisy ACE Algorithm and Multi-Layer Residual Learning.

Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint

PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks

Functional Adaptive Huber Linear Regression

Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks

Modeling Noisy Annotations for Point-wise Supervision