The study of loss function distributions is critical to characterize a
model's behaviour on a given machine learning problem. For example, while the
quality of a model is commonly determined by the average loss assessed on a
testing set, this quantity does not reflect the existence of the true mean of
the loss distribution. Indeed, the finiteness of the statistical moments of the
loss distribution is related to the thickness of its tails, which are generally
unknown. Since typical cross-validation schemes determine a family of testing
loss distributions conditioned on the training samples, the total loss
distribution must be recovered by marginalizing over the space of training
sets. As we show in this work, the finiteness of the sampling procedure
negatively affects the reliability and efficiency of classical tail estimation
methods from the Extreme Value Theory, such as the Peaks-Over-Threshold
approach. In this work we tackle this issue by developing a novel general
theory for estimating the tails of marginal distributions, when there exists a
large variability between locations of the individual conditional distributions
underlying the marginal. To this end, we demonstrate that under some regularity
conditions, the shape parameter of the marginal distribution is the maximum
tail shape parameter of the family of conditional distributions. We term this
estimation approach as Cross Tail Estimation (CTE). We test cross-tail
estimation in a series of experiments on simulated and real data, showing the
improved robustness and quality of tail estimation as compared to classical
approaches, and providing evidence for the relationship between overfitting and
loss distribution tail thickness.

本文提出了一种新的估计极值尾的方法 - CTE，相比于传统方法更为鲁棒，适用于带有大量变异的样本，同时通过实验研究证明了过度拟合与损失分布尾厚度之间的关系。