Conformal prediction has emerged as a rigorous means of providing deep
learning models with reliable uncertainty estimates and safety guarantees. Yet,
its performance is known to degrade under distribution shift and long-tailed
class distributions, which are often present in real world applications. Here,
we characterize the performance of several post-hoc and training-based
conformal prediction methods under these settings, providing the first
empirical evaluation on large-scale datasets and models. We show that across
numerous conformal methods and neural network families, performance greatly
degrades under distribution shifts violating safety guarantees. Similarly, we
show that in long-tailed settings the guarantees are frequently violated on
many classes. Understanding the limitations of these methods is necessary for
deployment in real world and safety-critical applications.

本文利用大规模数据集和模型对多种 conformal 方法和神经网络家族的性能进行了实际评估，证明了这些方法在分布转移和长尾设置下的性能严重下降，需要了解这些方法的局限性以在实际应用和安全关键应用中部署

验证在分布转移和长尾数据下的现代视觉架构上的符合预测

Empirically Validating Conformal Prediction on Modern Vision  Architectures Under Distribution Shift and Long-tailed Data

Long-tailed class distributions are prevalent among the practical
applications of object detection and instance segmentation. Prior work in
long-tail instance segmentation addresses the imbalance of losses between rare
and frequent categories by reducing the penalty for a model incorrectly
predicting a rare class label. We demonstrate that the rare categories are
heavily suppressed by correct background predictions, which reduces the
probability for all foreground categories with equal weight. Due to the
relative infrequency of rare categories, this leads to an imbalance that biases
towards predicting more frequent categories. Based on this insight, we develop
DropLoss -- a novel adaptive loss to compensate for this imbalance without a
trade-off between rare and frequent categories. With this loss, we show
state-of-the-art mAP across rare, common, and frequent categories on the LVIS
dataset.

本文提出一种新的自适应损失（DropLoss），用于解决目标检测中长尾类别分布下的损失不平衡问题，以改善罕见类别的预测结果，提高模型在 LVIS 数据集上的表现。

长尾实例分割的 DropLoss

DropLoss for Long-Tail Instance Segmentation

We present a new loss function called Distribution-Balanced Loss for the
multi-label recognition problems that exhibit long-tailed class distributions.
Compared to conventional single-label classification problem, multi-label
recognition problems are often more challenging due to two significant issues,
namely the co-occurrence of labels and the dominance of negative labels (when
treated as multiple binary classification problems). The Distribution-Balanced
Loss tackles these issues through two key modifications to the standard binary
cross-entropy loss: 1) a new way to re-balance the weights that takes into
account the impact caused by label co-occurrence, and 2) a negative tolerant
regularization to mitigate the over-suppression of negative labels. Experiments
on both Pascal VOC and COCO show that the models trained with this new loss
function achieve significant performance gains over existing methods. Code and
models are available at: this https URL .

本文提出了一种称为分布平衡损失函数的新方法，用于解决多标签识别问题中遇到的长尾类分布、标签共现和负标签占主导的问题，并通过实验验证了该方法的有效性。