Abstract:Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a U-shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count p grows past sample size n - a phenomenon dubbed double descent. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting. In this work, we take a closer look at evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold p=n. We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples, using which we find that their apparent double descent curves indeed fold back into more traditional convex shapes - providing a resolution to tensions between double descent and statistical intuition.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is whether the traditional understanding of the relationship between machine - learning model complexity and prediction error needs to be re - examined. Specifically, the paper challenges whether the "double - descent" phenomenon truly extends the limits of the traditional U - shaped complexity - generalization curve. By re - analyzing existing experimental evidence and re - thinking the method of parameter counting from the perspective of classical statistics, the paper explores whether the double - descent behavior observed in non - deep - learning models really contradicts the traditional U - shaped trade - off between model complexity and prediction error.
### Paper Background
Traditional statistical learning theory has established a widely understood relationship, that is, the relationship between model complexity and prediction error usually presents as a U - shaped curve, reflecting the transition from under - fitting to over - fitting. However, in recent years, due to the success of over - parameterized neural networks, some studies have proposed that this theory may be incomplete and introduced a new stage. That is, when the number of model parameters \( p \) exceeds the number of samples \( n \), the test error will experience a second descent, and this phenomenon is called "double - descent".
### Main Contributions
1. **Re - examining Existing Experimental Evidence**:
- The paper points out that the double - descent phenomenon observed in non - deep - learning methods (such as trees, boosting, and linear regression) is actually due to the implicit transformation that occurs when the number of parameters is increased on different complexity axes. Specifically, when the number of parameters exceeds a certain threshold, the model type changes, resulting in the second descent.
- For example, in a tree model, when the number of leaf nodes reaches the maximum, in order to further increase the number of parameters, it is necessary to increase the number of trees. This is actually the process of transforming from a single tree to a random forest.
2. **Re - thinking Parameter Counting from the Perspective of Classical Statistics**:
- The paper proposes a generalized measure of the effective number of parameters \( \hat{p_0} \) to measure the complexity of the model on unseen data. Using this measure, the paper finds that the so - called double - descent curve can actually be folded back into a more traditional U - shaped curve.
- This finding indicates that the double - descent phenomenon is not due to the increase in the number of parameters itself, but because the model has an essential change at the interpolation threshold \( p = n \).
### Experimental Analysis
- **Tree Model**:
- When the number of trees \( P_{\text{ens}} \) is fixed, the test error presents a U - shaped or L - shaped curve as the number of leaf nodes \( P_{\text{leaf}} \) increases.
- When the number of leaf nodes \( P_{\text{leaf}} \) is fixed, the test error presents an L - shaped curve as the number of trees \( P_{\text{ens}} \) increases.
- Only when the increase shifts from \( P_{\text{leaf}} \) to \( P_{\text{ens}} \) will the double - descent curve appear.
- **Gradient Boosting**:
- Similarly, when the number of trees \( P_{\text{ens}} \) is fixed, the test error presents a U - shaped or L - shaped curve as the number of boosting rounds \( P_{\text{boost}} \) increases.
- When the number of boosting rounds \( P_{\text{boost}} \) is fixed, the test error presents an L - shaped curve as the number of trees \( P_{\text{ens}} \) increases.
- When the increase shifts from \( P_{\text{boost}} \) to \( P_{\text{ens}} \), the double - descent curve also appears.
- **Linear Regression**:
- Use random Fourier features (RFF) to control the number of model parameters \( P_\phi \).
- When \( P_\phi \leq n \), the number of model parameters \( P_{\text{PC}} \) increases monotonically, and \( P_{\text{ex}} = 0 \).
- When \( P_\phi > n \), \( P_{\text{PC}} \) is fixed at \( n \), and \( P_{\text{ex}} \) increases monotonically.