Recent advances in visual anomaly detection research have seen AUROC and
AUPRO scores on public benchmark datasets such as MVTec and VisA converge
towards perfect recall, giving the impression that these benchmarks are
near-solved. However, high AUROC and AUPRO scores do not always reflect
qualitative performance, which limits the validity of these metrics in
real-world applications. We argue that the artificial ceiling imposed by the
lack of an adequate evaluation metric restrains progression of the field, and
it is crucial that we revisit the evaluation metrics used to rate our
algorithms. In response, we introduce Per-IMage Overlap (PIMO), a novel metric
that addresses the shortcomings of AUROC and AUPRO. PIMO retains the
recall-based nature of the existing metrics but introduces two distinctions:
the assignment of curves (and respective area under the curve) is per-image,
and its X-axis relies solely on normal images. Measuring recall per image
simplifies instance score indexing and is more robust to noisy annotations. As
we show, it also accelerates computation and enables the usage of statistical
tests to compare models. By imposing low tolerance for false positives on
normal images, PIMO provides an enhanced model validation procedure and
highlights performance variations across datasets. Our experiments demonstrate
that PIMO offers practical advantages and nuanced performance insights that
redefine anomaly detection benchmarks -- notably challenging the perception
that MVTec AD and VisA datasets have been solved by contemporary models.
Available on GitHub: this https URL

最近在视觉异常检测研究中取得的进展看到在公共基准数据集 (MVTec 和 VisA) 上的 AUROC 和 AUPRO 得分收敛于完美的召回率，给人一种这些基准已经接近解决的印象。然而，高 AUROC 和 AUPRO 得分并不总能反映出定性性能，这限制了这些度量在实际应用中的有效性。我们认为，由于缺乏适当的评估度量所带来的人为限制制约了该领域的进展，并且有必要重新审视用于评估算法的评估度量。为此，我们引入了 Per-IMage Overlap (PIMO)，一种新颖的度量方法，解决了 AUROC 和 AUPRO 的不足之处。PIMO 保留了现有度量的基于召回率的特性，但引入了两个区别：曲线 (以及相应的曲线下面积) 的分配是以图像为单位的，而其 X 轴仅依赖于正常图像。通过每幅图像测量召回率，简化了实例分数索引并对噪声注释更加健壮。正如我们所展示的，它还加速了计算并使我们能够使用统计测试来比较模型。通过对正常图像施加低的误报容忍度，PIMO 提供了一个增强的模型验证过程，并突显了跨数据集的性能变化。我们的实验证明，PIMO 提供了实际的优势和细致的性能见解，重新定义了异常检测基准，特别是挑战了现代模型解决了 MVTec AD 和 VisA 数据集的看法。