Research on Out-Of-Distribution (OOD) detection focuses mainly on building
scores that efficiently distinguish OOD data from In Distribution (ID) data. On
the other hand, Conformal Prediction (CP) uses non-conformity scores to
construct prediction sets with probabilistic coverage guarantees. In this work,
we propose to use CP to better assess the efficiency of OOD scores.
Specifically, we emphasize that in standard OOD benchmark settings, evaluation
metrics can be overly optimistic due to the finite sample size of the test
dataset. Based on the work of (Bates et al., 2022), we define new conformal
AUROC and conformal FRP@TPR95 metrics, which are corrections that provide
probabilistic conservativeness guarantees on the variability of these metrics.
We show the effect of these corrections on two reference OOD and anomaly
detection benchmarks, OpenOOD (Yang et al., 2022) and ADBench (Han et al.,
2022). We also show that the benefits of using OOD together with CP apply the
other way around by using OOD scores as non-conformity scores, which results in
improving upon current CP methods. One of the key messages of these
contributions is that since OOD is concerned with designing scores and CP with
interpreting these scores, the two fields may be inherently intertwined.

提供了一种使用 Conformal Prediction（CP）来更好评估 OOD 分数的方法，并通过引入新的 conformal AUROC 和 conformal FRP@TPR95 度量对 OOD 基准进行了改进，同时展示了 OOD 和 CP 之间相辅相成的优势。

外部分布检测应该使用符合性预测（反之亦然？）

Out-of-Distribution Detection Should Use Conformal Prediction (and  Vice-versa?)

Numerous pre-training techniques for visual document understanding (VDU) have
recently shown substantial improvements in performance across a wide range of
document tasks. However, these pre-trained VDU models cannot guarantee
continued success when the distribution of test data differs from the
distribution of training data. In this paper, to investigate how robust
existing pre-trained VDU models are to various distribution shifts, we first
develop an out-of-distribution (OOD) benchmark termed Do-GOOD for the
fine-Grained analysis on Document image-related tasks specifically. The Do-GOOD
benchmark defines the underlying mechanisms that result in different
distribution shifts and contains 9 OOD datasets covering 3 VDU related tasks,
e.g., document information extraction, classification and question answering.
We then evaluate the robustness and perform a fine-grained analysis of 5 latest
VDU pre-trained models and 2 typical OOD generalization algorithms on these OOD
datasets. Results from the experiments demonstrate that there is a significant
performance gap between the in-distribution (ID) and OOD settings for document
images, and that fine-grained analysis of distribution shifts can reveal the
brittle nature of existing pre-trained VDU models and OOD generalization
algorithms. The code and datasets for our Do-GOOD benchmark can be found at
this https URL

本研究基于文档图像相关任务，开发了一个名为 Do-GOOD 的 OOD 基准，探究现有预训练 VDU 模型对各种分布转移的鲁棒性及其脆弱性。实验结果发现，在文档图像领域，现有预训练 VDU 模型以及 OOD 泛化算法在分布转移方面表现出较大的性能差距和脆弱性。