Abstract:Despite their overwhelming capacity to overfit, deep neural networks trained by specific optimization algorithms tend to generalize relatively well to unseen data. Recently, researchers explained it by investigating the implicit bias of optimization algorithms. A remarkable progress is the work (Lyu & Li, 2019), which proves gradient descent (GD) maximizes the margin of homogeneous deep neural networks. Except the first-order optimization algorithms like GD, adaptive algorithms such as AdaGrad, RMSProp and Adam are popular owing to their rapid training process. Mean-while, numerous works have provided empirical evidence that adaptive methods may suffer from poor generalization performance. However, theoretical explanation for the generalization of adaptive optimization algorithms is still lacking. In this paper, we study the implicit bias of adaptive optimization algorithms on homogeneous neural networks. In particular, we study the convergent direction of parameters when they are optimizing the logistic loss. We prove that the convergent direction of Adam and RMSProp is the same as GD, while for AdaGrad, the convergent direction depends on the adaptive conditioner. Technically, we provide a unified framework to analyze convergent direction of adaptive optimization algorithms by constructing novel and nontrivial adaptive gradient flow and surrogate margin. The theoretical findings explain the superiority on generalization of exponential moving average strategy that is adopted by RMSProp and Adam. To the best of knowledge, it is the first work to study the convergent direction of adaptive optimizations on non-linear deep neural …

On the Implicit Bias of Linear Equivariant Steerable Networks

The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks

Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks

The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks

How to induce regularization in linear models: A guide to reparametrizing gradient flow

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data

Rethinking the Benefits of Steerable Features in 3D Equivariant Graph Neural Networks

Implicit Bias of Mirror Descent for Shallow Neural Networks in Univariate Regression

The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program

A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs

A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks

The Implicit Bias of AdaGrad on Separable Data

Intertwiners between Induced Representations (with Applications to the Theory of Equivariant Neural Networks)

The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks

Lie Group Decompositions for Equivariant Neural Networks

Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups

Linear Convergence of Adaptive Stochastic Gradient Descent

Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training

Nonasymptotic theory for two-layer neural networks: Beyond the bias-variance trade-off

The implicit bias for adaptive optimization algorithms on homogeneous neural networks

Optimization Insights into Deep Diagonal Linear Networks