Abstract:Second-order optimization has been shown to accelerate the training of deep neural networks in many applications, often yielding faster progress per iteration on the training loss compared to first-order <a class="link-external link-http" href="http://optimizers.However" rel="external noopener nofollow">this http URL</a>, the generalization properties of second-order methods are still being debated. Theoretical investigations have proved difficult to carry out outside the tractable settings of heavily simplified model classes -- thus, the relevance of existing theories to practical deep learning applications remains unclear. Similarly, empirical studies in large-scale models and real datasets are significantly confounded by the necessity to approximate second-order updates in practice. It is often unclear whether the observed generalization behaviour arises specifically from the second-order nature of the parameter updates, or instead reflects the specific structured (e.g.\ Kronecker) approximations used or any damping-based interpolation towards first-order updates. Here, we show for the first time that exact Gauss-Newton (GN) updates take on a tractable form in a class of deep reversible architectures that are sufficiently expressive to be meaningfully applied to common benchmark datasets. We exploit this novel setting to study the training and generalization properties of the GN optimizer. We find that exact GN generalizes poorly. In the mini-batch training setting, this manifests as rapidly saturating progress even on the \emph{training} loss, with parameter updates found to overfit each mini-batchatch without producing the features that would support generalization to other mini-batches. We show that our experiments run in the ``lazy'' regime, in which the neural tangent kernel (NTK) changes very little during the course of training. This behaviour is associated with having no significant changes in neural representations, explaining the lack of generalization.

Generalization Ability of Universal Learning Network by Using Second Order Derivatives

Improving Generalization Ability of Universal Learning Networks with Superfluous Parameters

Generalization Ability of Modeling Dynamic Systems Using Universal Learning Network

Generalization Ability of Universal Learning Network by Optimizing Network Size and Time Delay.

Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory

Generalization Error Analysis of Neural networks with Gradient Based Regularization

Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization

General Backpropagation Algorithm for Training Second-order Neural Networks

On improving generalization in a class of learning problems with the method of small parameters for weakly-controlled optimal gradient systems

Second-order Neural Network Training Using Complex-step Directional Derivative

Understanding Why Neural Networks Generalize Well Through GSNR of Parameters

Regularization theory in the study of generalization ability of a biological neural network model

Second-Order Neural ODE Optimizer

A new regularization learning method for improving generalization capability of neural network

A Kind of Second-Order Learning Algorithm Based on Generalized Cost Criteria in Multi-Layer Feed-Forward Neural Networks

A Comparative Analysis of Optimization and Generalization Properties of Two-Layer Neural Network and Random Feature Models under Gradient Descent Dynamics

Numerical Differentiation for the Second Order Derivatives of Functions of Two Variables

A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

A Fuzzy Optimization Neural Network Model Using Second Order Information

Neural Generalized Ordinary Differential Equations with Layer-varying Parameters

Modeling Nonlinear Dynamic Systems Using Universal Learning Network with Filtering Mechanism