Abstract:Overparameterized models may have many interpolating solutions; implicit regularization refers to the hidden preference of a particular optimization method towards a certain interpolating solution among the many. A by now established line of work has shown that (stochastic) gradient descent tends to have an implicit bias towards low rank and/or sparse solutions when used to train deep linear networks, explaining to some extent why overparameterized neural network models trained by gradient descent tend to have good generalization performance in practice. However, existing theory for square-loss objectives often requires very small initialization of the trainable weights, which is at odds with the larger scale at which weights are initialized in practice for faster convergence and better generalization performance. In this paper, we aim to close this gap by incorporating and analyzing gradient flow (continuous-time version of gradient descent) with weight normalization, where the weight vector is reparameterized in terms of polar coordinates, and gradient flow is applied to the polar coordinates. By analyzing key invariants of the gradient flow and using Lojasiewicz Theorem, we show that weight normalization also has an implicit bias towards sparse solutions in the diagonal linear model, but that in contrast to plain gradient flow, weight normalization enables a robust bias that persists even if the weights are initialized at practically large scale. Experiments suggest that the gains in both convergence speed and robustness of the implicit bias are improved dramatically by using weight normalization in overparameterized diagonal linear network models.

A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks

Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations

The Efficacy of Regularization in Two Layer Neural Networks

Robust Implicit Regularization via Weight Normalization

Learning Continually by Spectral Regularization

Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel

Implicit Regularization via Spectral Neural Networks and Non-linear Matrix Sensing

Generalizing and Improving Jacobian and Hessian Regularization

Function-Space Optimality of Neural Architectures With Multivariate Nonlinearities

Orthogonal Transforms in Neural Networks Amount to Effective Regularization

Nonasymptotic theory for two-layer neural networks: Beyond the bias-variance trade-off

Scalable Optimization in the Modular Norm

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systems

Function-Space Regularization in Neural Networks: A Probabilistic Perspective

Breaking the Curse of Dimensionality with Convex Neural Networks

Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network

Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients

Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalization

Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay