Abstract:The study of loss function distributions is critical to characterize a model's behaviour on a given machine learning problem. For example, while the quality of a model is commonly determined by the average loss assessed on a testing set, this quantity does not reflect the existence of the true mean of the loss distribution. Indeed, the finiteness of the statistical moments of the loss distribution is related to the thickness of its tails, which are generally unknown. Since typical cross-validation schemes determine a family of testing loss distributions conditioned on the training samples, the total loss distribution must be recovered by marginalizing over the space of training sets. As we show in this work, the finiteness of the sampling procedure negatively affects the reliability and efficiency of classical tail estimation methods from the Extreme Value Theory, such as the Peaks-Over-Threshold approach. In this work we tackle this issue by developing a novel general theory for estimating the tails of marginal distributions, when there exists a large variability between locations of the individual conditional distributions underlying the marginal. To this end, we demonstrate that under some regularity conditions, the shape parameter of the marginal distribution is the maximum tail shape parameter of the family of conditional distributions. We term this estimation approach as Cross Tail Estimation (CTE). We test cross-tail estimation in a series of experiments on simulated and real data, showing the improved robustness and quality of tail estimation as compared to classical approaches, and providing evidence for the relationship between overfitting and loss distribution tail thickness.

Where does the tail start? Inflection Points and Maximum Curvature as Boundaries

The multivariate tail-inflated normal distribution and its application in finance

On Tail Decay Rate Estimation of Loss Function Distributions

Tail behaviour of stationary densities for one-dimensional random diffeomorphisms

Detecting tail behavior: mean excess plots with confidence bounds

A Method For Bounding Tail Probabilities

Extreme and Inference for Tail Gini Functionals with Applications in Tail Risk Measurement

Nonparametric Inference for Distortion Risk Measures on Tail Regions

Intermediate Tail Dependence: A Review and Some New Results

Tail diversity from inflation

Tail Index Estimation for Discrete Heavy-Tailed Distributions

Mastering the Body and Tail Shape of a Distribution

Entropic Plots for Tail Type Classification

Sharp upper tail behavior of line ensembles via the tangent method

Bridging Extremes: The Invertible Bimodal Gumbel Distribution

A New Type Of Upper And Lower Bounds On Right-Tail Probabilities Of Continuous Random Variables

Distributional Tail Estimation Through Its Characteristic Function

Flexible Tails for Normalising Flows, with Application to the Modelling of Financial Return Data

On tail inference in iid settings with nonnegative extreme value index

Tail estimates for the Brownian excursion area and other Brownian areas

The geometry of the maximum likelihood of Cauchy-like distributions