Abstract: Deep neural networks with batch normalization (BN-DNNs) are invariant to weight rescaling due to their normalization operations. However, using weight decay (WD) benefits these weight-scale-invariant networks, which is often attributed to an increase of the effective learning rate when the weight norms are decreased. In this paper, we demonstrate the insufficiency of the previous explanation and investigate the implicit biases of stochastic gradient descent (SGD) on BN-DNNs to provide a theoretical explanation for the efficacy of weight decay. We identity two implicit biases of SGD on BN-DNNs: 1) the weight norms in SGD training remain constant in the continuous-time domain and keep increasing in the discrete-time domain; 2) SGD optimizes weight vectors in fully-connected networks or convolution kernels in convolution neural networks by updating components lying in the input feature span, while leaving those components orthogonal to the input feature span unchanged. Thus, SGD without WD accumulates weight noise orthogonal to the input feature span, and cannot eliminate such noise. Our empirical studies corroborate the hypothesis that weight decay suppresses weight noise that is left untouched by SGD. Furthermore, we propose to use weight rescaling (WRS) instead of weight decay to achieve the same regularization effect, while avoiding performance degradation of WD on some momentum-based optimizers. Our empirical results on image recognition show that regardless of optimization methods and network architectures, training BN-DNNs using WRS achieves similar or better performance compared with using WD. We also show that training with WRS generalizes better compared to WD, on other computer vision tasks.

Delving into Variance Transmission and Normalization: Shift of Average Gradient Makes the Network Collapse

Generalized Batch Normalization: Towards Accelerating Deep Neural Networks

Normalized Activation Function: Toward Better Convergence

Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift.

Why Batch Normalization Works? A Buckling Perspective

New Interpretations of Normalization Methods in Deep Learning.

A Mean Field Theory of Batch Normalization

The Implicit Biases of Stochastic Gradient Descent on Deep Neural Networks with Batch Normalization

How Does BN Increase Collapsed Neural Network Filters?

On Centralization and Unitization of Batch Normalization for Deep ReLU Neural Networks

Towards Understanding Regularization in Batch Normalization

Diminishing Batch Normalization

BNET: Batch Normalization with Enhanced Linear Transformation.

Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

Iterative Normalization: Beyond Standardization towards Efficient Whitening

Understanding the Disharmony between Weight Normalization Family and Weight Decay: shifted Regularizer

Normalization of Neural Networks using Analytic Variance Propagation

Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalization

Delving into the Estimation Shift of Batch Normalization in a Network

Stochastic Whitening Batch Normalization