Detecting test-time distribution shift has emerged as a key capability for safely deployed machine learning models, with the question being tackled under various guises in recent years. In this paper, we aim to provide a consolidated view of the two largest sub-fields within the community: out-of-distribution (OOD) detection and open-set recognition (OSR). In particular, we aim to provide rigorous empirical analysis of different methods across settings and provide actionable takeaways for practitioners and researchers. Concretely, we make the following contributions: (i) We perform rigorous cross-evaluation between state-of-the-art methods in the OOD detection and OSR settings and identify a strong correlation between the performances of methods for them; (ii) We propose a new, large-scale benchmark setting which we suggest better disentangles the problem tackled by OOD detection and OSR, re-evaluating state-of-the-art OOD detection and OSR methods in this setting; (iii) We surprisingly find that the best performing method on standard benchmarks (Outlier Exposure) struggles when tested at scale, while scoring rules which are sensitive to the deep feature magnitude consistently show promise; and (iv) We conduct empirical analysis to explain these phenomena and highlight directions for future research. Code: https://github.com/Visual-AI/Dissect-OOD-OSR

本研究针对机器学习模型在测试时的分布转移问题进行了深入分析，特别是分布外检测和开放集识别两个领域。通过对现有方法的严格交叉评估，我们发现它们之间存在显著的性能相关性，并提出了一个新的基准设置，以更好地解耦这两个问题。研究结果表明，传统多项基准中表现最好的方法在大规模测试时存在困难，而对深度特征大小敏感的评分规则显示出一致的潜力，指明了未来研究的方向。

离散化分布外检测与开放集识别：方法与基准的关键分析