Abstract:The study of loss function distributions is critical to characterize a model's behaviour on a given machine learning problem. For example, while the quality of a model is commonly determined by the average loss assessed on a testing set, this quantity does not reflect the existence of the true mean of the loss distribution. Indeed, the finiteness of the statistical moments of the loss distribution is related to the thickness of its tails, which are generally unknown. Since typical cross-validation schemes determine a family of testing loss distributions conditioned on the training samples, the total loss distribution must be recovered by marginalizing over the space of training sets. As we show in this work, the finiteness of the sampling procedure negatively affects the reliability and efficiency of classical tail estimation methods from the Extreme Value Theory, such as the Peaks-Over-Threshold approach. In this work we tackle this issue by developing a novel general theory for estimating the tails of marginal distributions, when there exists a large variability between locations of the individual conditional distributions underlying the marginal. To this end, we demonstrate that under some regularity conditions, the shape parameter of the marginal distribution is the maximum tail shape parameter of the family of conditional distributions. We term this estimation approach as Cross Tail Estimation (CTE). We test cross-tail estimation in a series of experiments on simulated and real data, showing the improved robustness and quality of tail estimation as compared to classical approaches, and providing evidence for the relationship between overfitting and loss distribution tail thickness.

Tail Estimation of the Spectral Density under Fixed-Domain Asymptotics

Fluctuations of Deformed Wigner Random Matrices

Gaussian Tail for Empirical Distributions of MST on Random Graphs

Approximation of the Tail Probability of Randomly Weighted Sums and Applications

High-Frequency Tail Index Estimation by Nearly Tight Frames

Tail Index Varying Coefficient Model

On Asymptotic Properties of the Parameter Estimator for a Type of SPDE

On asymptotics of eigenvectors of large sample covariance matrix

Asymptotics of Spectral Density Estimates

Tail behaviour of stationary densities for one-dimensional random diffeomorphisms

Tail Bounds on the Spectral Norm of Sub-Exponential Random Matrices

Tail Asymptotics for the Extremes of Bivariate Gaussian Random Fields

Quantitative Error Analyses of Spectral Density Estimators Using Covariance Lags

Asymptotic Spectral Theory for Spatial Data

Tail Index Estimation for Discrete Heavy-Tailed Distributions

Needlet-Whittle Estimates on the Unit Sphere

On Tail Decay Rate Estimation of Loss Function Distributions

Assessing Tail Risk Using Expectile Regressions with Partially Varying Coefficients

Tail estimates for homogenization theorems in random media

Exponential and Double Exponential Tails for Maximum of Two-Dimensional Discrete Gaussian Free Field

Tail index estimation for tail adversarial stable time series with an application to high‐dimensional tail clustering