Abstract:Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a U-shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count p grows past sample size n - a phenomenon dubbed double descent. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting. In this work, we take a closer look at evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold p=n. We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples, using which we find that their apparent double descent curves indeed fold back into more traditional convex shapes - providing a resolution to tensions between double descent and statistical intuition.

Manipulating Sparse Double Descent

Sparse Double Descent: Where Network Pruning Aggravates Overfitting

Regularization-wise double descent: Why it occurs and how to eliminate it

Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle

Can we avoid Double Descent in Deep Neural Networks?

DSD$^2$: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?

Understanding the Role of Optimization in Double Descent

Multi-scale Feature Learning Dynamics: Insights for Double Descent

Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks

Phenomenology of Double Descent in Finite-Width Neural Networks

Understanding the Double Descent Phenomenon in Deep Learning

When and how epochwise double descent happens

Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space

Dropout Drops Double Descent

Does Double Descent Occur in Self-Supervised Learning?

On the Lipschitz Constant of Deep Networks and Double Descent

Sparse Double Descent in Vision Transformers: real or phantom threat?

Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenon.

The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks

A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

Implicit Sparse Regularization: The Impact of Depth and Early Stopping