Abstract:The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting. We survey recent theoretical progress that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behavior of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favorable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

A Kind of Dependable Learning Method Based on Cogradient Descent

How far from automatically interpreting deep learning

Interpreting Deep Learning Model Using Rule-based Method

Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias

SSGD: A Safe and Efficient Method of Gradient Descent

Learning Gradient Descent: Better Generalization and Longer Horizons

Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation

Gradient Correction Beyond Gradient Descent

Understanding the black-box: towards interpretable and reliable deep learning models

Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle

Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy

Transforming gradient-based techniques into interpretable methods

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Towards interpretable-by-design deep learning algorithms

Gradient Descent based Optimization Algorithms for Deep Learning Models Training

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

Deep learning: a statistical viewpoint

Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks

PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks

On Gradient-like Explanation under a Black-box Setting: When Black-box Explanations Become as Good as White-box

A General Descent Aggregation Framework for Gradient-based Bi-level Optimization