Abstract:Second-order optimization has been shown to accelerate the training of deep neural networks in many applications, often yielding faster progress per iteration on the training loss compared to first-order <a class="link-external link-http" href="http://optimizers.However" rel="external noopener nofollow">this http URL</a>, the generalization properties of second-order methods are still being debated. Theoretical investigations have proved difficult to carry out outside the tractable settings of heavily simplified model classes -- thus, the relevance of existing theories to practical deep learning applications remains unclear. Similarly, empirical studies in large-scale models and real datasets are significantly confounded by the necessity to approximate second-order updates in practice. It is often unclear whether the observed generalization behaviour arises specifically from the second-order nature of the parameter updates, or instead reflects the specific structured (e.g.\ Kronecker) approximations used or any damping-based interpolation towards first-order updates. Here, we show for the first time that exact Gauss-Newton (GN) updates take on a tractable form in a class of deep reversible architectures that are sufficiently expressive to be meaningfully applied to common benchmark datasets. We exploit this novel setting to study the training and generalization properties of the GN optimizer. We find that exact GN generalizes poorly. In the mini-batch training setting, this manifests as rapidly saturating progress even on the \emph{training} loss, with parameter updates found to overfit each mini-batchatch without producing the features that would support generalization to other mini-batches. We show that our experiments run in the ``lazy'' regime, in which the neural tangent kernel (NTK) changes very little during the course of training. This behaviour is associated with having no significant changes in neural representations, explaining the lack of generalization.

Efficient Second-Order Optimization for Neural Networks with Kernel Machines.

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Second-order Neural Network Training Using Complex-step Directional Derivative

Scalable Second Order Optimization for Deep Learning

Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization

SGD with Partial Hessian for Deep Neural Networks Optimization

Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation

Optimizing Kernel Machines using Deep Learning

Sparse Optimization for Second Order Kernel Methods

Block-diagonal Hessian-free Optimization for Training Neural Networks

Efficient Second-Order Neural Network Optimization via Adaptive Trust Region Methods

Minibatching Offers Improved Generalization Performance for Second Order Optimizers

Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations

Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent

Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient

Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks

Exact Gauss-Newton Optimization for Training Deep Neural Networks

Eva: A General Vectorized Approximation Framework for Second-order Optimization

Fast Second-Order Online Kernel Learning through Incremental Matrix Sketching and Decomposition