Abstract:The study of loss function distributions is critical to characterize a model's behaviour on a given machine learning problem. For example, while the quality of a model is commonly determined by the average loss assessed on a testing set, this quantity does not reflect the existence of the true mean of the loss distribution. Indeed, the finiteness of the statistical moments of the loss distribution is related to the thickness of its tails, which are generally unknown. Since typical cross-validation schemes determine a family of testing loss distributions conditioned on the training samples, the total loss distribution must be recovered by marginalizing over the space of training sets. As we show in this work, the finiteness of the sampling procedure negatively affects the reliability and efficiency of classical tail estimation methods from the Extreme Value Theory, such as the Peaks-Over-Threshold approach. In this work we tackle this issue by developing a novel general theory for estimating the tails of marginal distributions, when there exists a large variability between locations of the individual conditional distributions underlying the marginal. To this end, we demonstrate that under some regularity conditions, the shape parameter of the marginal distribution is the maximum tail shape parameter of the family of conditional distributions. We term this estimation approach as Cross Tail Estimation (CTE). We test cross-tail estimation in a series of experiments on simulated and real data, showing the improved robustness and quality of tail estimation as compared to classical approaches, and providing evidence for the relationship between overfitting and loss distribution tail thickness.

Detecting tail behavior: mean excess plots with confidence bounds

Empirical Likelihood Based Confidence Intervals For The Tail Index When Gamma <-1/2

Empirical Likelihood Based Inference for Conditional Pareto-type Tail Index

Divergence based robust estimation of the tail index through an exponential regression model

Robust estimation of Pareto-type tail index through an exponential regression model

Fitting tails affected by truncation

On tail inference in iid settings with nonnegative extreme value index

Entropic Plots for Tail Type Classification

On Tail Decay Rate Estimation of Loss Function Distributions

Tail Behavior of Over-Dispersed Counts Distributions

Estimation of the conditional tail moment for Weibull‐type distributions

Tail bounds for empirically standardized sums

Bias-reduced and variance-corrected asymptotic Gaussian inference about extreme expectiles

Extreme and Inference for Tail Gini Functionals with Applications in Tail Risk Measurement

Modeling spatial extremes using normal mean-variance mixtures

The Efficient Tail Hypothesis: An Extreme Value Perspective on Market Efficiency

Mean estimation and regression under heavy-tailed distributions--a survey

Extreme expectile estimation for short-tailed data

Where does the tail start? Inflection Points and Maximum Curvature as Boundaries

A Study on Tail Behavior for Stock Returns Distribution

Examining Tail Index Estimators in New Pareto Distribution: Monte Carlo Simulations and Income Data Applications