Abstract:Second-order optimization has been shown to accelerate the training of deep neural networks in many applications, often yielding faster progress per iteration on the training loss compared to first-order <a class="link-external link-http" href="http://optimizers.However" rel="external noopener nofollow">this http URL</a>, the generalization properties of second-order methods are still being debated. Theoretical investigations have proved difficult to carry out outside the tractable settings of heavily simplified model classes -- thus, the relevance of existing theories to practical deep learning applications remains unclear. Similarly, empirical studies in large-scale models and real datasets are significantly confounded by the necessity to approximate second-order updates in practice. It is often unclear whether the observed generalization behaviour arises specifically from the second-order nature of the parameter updates, or instead reflects the specific structured (e.g.\ Kronecker) approximations used or any damping-based interpolation towards first-order updates. Here, we show for the first time that exact Gauss-Newton (GN) updates take on a tractable form in a class of deep reversible architectures that are sufficiently expressive to be meaningfully applied to common benchmark datasets. We exploit this novel setting to study the training and generalization properties of the GN optimizer. We find that exact GN generalizes poorly. In the mini-batch training setting, this manifests as rapidly saturating progress even on the \emph{training} loss, with parameter updates found to overfit each mini-batchatch without producing the features that would support generalization to other mini-batches. We show that our experiments run in the ``lazy'' regime, in which the neural tangent kernel (NTK) changes very little during the course of training. This behaviour is associated with having no significant changes in neural representations, explaining the lack of generalization.

Minibatching Offers Improved Generalization Performance for Second Order Optimizers

Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization

Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradients

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Why Does Large Batch Training Result in Poor Generalization? A Comprehensive Explanation and a Better Strategy from the Viewpoint of Stochastic Optimization

Stability and Generalization for Minibatch SGD and Local SGD

A new perspective for understanding generalization gap of deep neural networks trained with large batch sizes

A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes

Mini-batch Quasi-Newton Optimization for Large Scale Linear Support Vector Regression

Optimization Variance: Exploring Generalization Properties of DNNs

Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithms

Efficient Second-Order Optimization for Neural Networks with Kernel Machines.

Stochastic batch size for adaptive regularization in deep network optimization

Efficient mini-batch training for stochastic optimization

Scalable Second Order Optimization for Deep Learning

Towards Better Generalization: BP-SVRG in Training Deep Neural Networks

Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models

AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

Demystifying SGD with Doubly Stochastic Gradients

Three Factors Influencing Minima in SGD

Asgd: Stochastic Gradient Descent with Adaptive Batch Size for Every Parameter