Uncertainty estimation is an essential and heavily-studied component for the reliable application of semantic segmentation methods. While various studies exist claiming methodological advances on the one hand, and successful application on the other hand, the field is currently hampered by a gap between theory and practice leaving fundamental questions unanswered: Can data-related and model-related uncertainty really be separated in practice? Which components of an uncertainty method are essential for real-world performance? Which uncertainty method works well for which application? In this work, we link this research gap to a lack of systematic and comprehensive evaluation of uncertainty methods. Specifically, we identify three key pitfalls in current literature and present an evaluation framework that bridges the research gap by providing 1) a controlled environment for studying data ambiguities as well as distribution shifts, 2) systematic ablations of relevant method components, and 3) test-beds for the five predominant uncertainty applications: OoD-detection, active learning, failure detection, calibration, and ambiguity modeling. Empirical results on simulated as well as real-world data demonstrate how the proposed framework is able to answer the predominant questions in the field revealing for instance that 1) separation of uncertainty types works on simulated data but does not necessarily translate to real-world data, 2) aggregation of scores is a crucial but currently neglected component of uncertainty methods, 3) While ensembles are performing most robustly across the different downstream tasks and settings, test-time augmentation often constitutes a light-weight alternative. Code is at: https://github.com/IML-DKFZ/values

在这项研究中，我们将当前研究中不确定性方法的三个关键问题与缺乏系统和全面的评估方法相联系，并提出了一种评估框架，通过提供1）用于研究数据模糊性和分布变化的受控环境，2）对相关方法组件进行系统性切割，以及3）用于五种主要的不确定性应用（OoD检测，主动学习，故障检测，校准和歧义建模）的测试基准，来弥补研究差距。对于模拟数据和真实世界数据的实证结果表明，所提出的框架能够回答该领域的主要问题，揭示了例如：1）不确定性类型的分离在模拟数据上是有效的，但不一定适用于真实数据，2）分数的聚合是不确定性方法中重要但目前被忽视的组件，3）尽管集成模型在不同下游任务和设置中表现最为稳健，但测试时间增强常常作为一种轻量级的替代方法。

ValUES: 语义分割中不确定性估计系统验证的框架