Abstract:Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a U-shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count p grows past sample size n - a phenomenon dubbed double descent. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting. In this work, we take a closer look at evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold p=n. We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples, using which we find that their apparent double descent curves indeed fold back into more traditional convex shapes - providing a resolution to tensions between double descent and statistical intuition.

Multiple Descent: Design Your Own Generalization Curve

Least Squares Regression Can Exhibit Under-Parameterized Double Descent

Multi-scale Feature Learning Dynamics: Insights for Double Descent

Generalization Error of Generalized Linear Models in High Dimensions

A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

On the Lipschitz Constant of Deep Networks and Double Descent

On the curvature of the loss landscape

Double Descent and Other Interpolation Phenomena in GANs

Phenomenology of Double Descent in Finite-Width Neural Networks

Regularization-wise double descent: Why it occurs and how to eliminate it

Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle

Understanding the Double Descent Phenomenon in Deep Learning

The twin peaks of learning neural networks

Learning Curves for Analysis of Deep Networks

Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression

Generalization by design: Shortcuts to Generalization in Deep Learning

Analysis of Interpolating Regression Models and the Double Descent Phenomenon

Understanding the Role of Optimization in Double Descent

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes.

Stochastic Gradient Descent Introduces an Effective Landscape-Dependent Regularization Favoring Flat Solutions