Abstract:When equipped with efficient optimization algorithms, the over-parameterized neural networks have demonstrated high level of performance even though the loss function is non-convex and non-smooth. While many works have been focusing on understanding the loss dynamics by training neural networks with the gradient descent (GD), in this work, we consider a broad class of optimization algorithms that are commonly used in practice. For example, we show from a dynamical system perspective that the Heavy Ball (HB) method can converge to global minimum on mean squared error (MSE) at a linear rate (similar to GD); however, the Nesterov accelerated gradient descent (NAG) may only converges to global minimum sublinearly. Our results rely on the connection between neural tangent kernel (NTK) and finite over-parameterized neural networks with ReLU activation, which leads to analyzing the limiting ordinary differential equations (ODE) for optimization algorithms. We show that, optimizing the non-convex loss over the weights corresponds to optimizing some strongly convex loss over the prediction error. As a consequence, we can leverage the classical convex optimization theory to understand the convergence behavior of neural networks. We believe our approach can also be extended to other optimization algorithms and network architectures.

Mathematical Models of Overparameterized Neural Networks

Mathematical models of overparameterized neural networks

Convex Formulation of Overparameterized Deep Neural Networks

Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks

Convergence Analysis for Over-Parameterized Deep Learning

Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Over-parameterization and Adversarial Robustness in Neural Networks: An Overview and Empirical Analysis

Towards a mathematical understanding of neural network-based machine learning: what we know and what we don't

Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality

Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks

How Does Overparameterization Affect Features?

Understanding over-parameterized deep networks by geometrization

A Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

Neural Networks for Parameter Estimation in Intractable Models

A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks

Advanced Deep Neural Network Technique for Microwave Parametric Modeling

Estimation of Over-parameterized Models from an Auto-Modeling Perspective

A Geometric Modeling of Occam's Razor in Deep Learning