Annotators exhibit disagreement during data labeling, which can be termed as
annotator label uncertainty. Annotator label uncertainty manifests in
variations of labeling quality. Training with a single low-quality annotation
per sample induces model reliability degradations. In this work, we first
examine the effects of annotator label uncertainty in terms of the model's
generalizability and prediction uncertainty. We observe that the model's
generalizability and prediction uncertainty degrade with the presence of
low-quality noisy labels. Meanwhile, our evaluation of existing uncertainty
estimation algorithms indicates their incapability in response to annotator
label uncertainty. To mitigate performance degradation, prior methods show that
training models with labels collected from multiple independent annotators can
enhance generalizability. However, they require massive annotations. Hence, we
introduce a novel perceptual quality-based model training framework to
objectively generate multiple labels for model training to enhance reliability,
while avoiding massive annotations. Specifically, we first select a subset of
samples with low perceptual quality scores ranked by statistical regularities
of visual signals. We then assign de-aggregated labels to each sample in this
subset to obtain a training set with multiple labels. Our experiments and
analysis demonstrate that training with the proposed framework alleviates the
degradation of generalizability and prediction uncertainty caused by annotator
label uncertainty.

本文研究了数据标注中的注释者标签不确定性对模型的泛化能力和预测不确定性的影响，并提出了一种基于感知质量的模型训练框架，通过生成多个标签来增强模型的可靠性。实验证明，使用该框架进行训练可以减轻注释者标签不确定性对模型泛化能力和预测不确定性的降低。

基于感知质量的模型训练在注释者标签不确定性下的应用

Perceptual Quality-based Model Training under Annotator Label  Uncertainty

The modern pervasiveness of large-scale deep neural networks (NNs) is driven
by their extraordinary performance on complex problems but is also plagued by
their sudden, unexpected, and often catastrophic failures, particularly on
challenging scenarios. Existing algorithms that provide risk-awareness to NNs
are complex and ad-hoc. Specifically, these methods require significant
engineering changes, are often developed only for particular settings, and are
not easily composable. Here we present capsa, a framework for extending models
with risk-awareness. Capsa provides a methodology for quantifying multiple
forms of risk and composing different algorithms together to quantify different
risk metrics in parallel. We validate capsa by implementing state-of-the-art
uncertainty estimation algorithms within the capsa framework and benchmarking
them on complex perception datasets. We demonstrate capsa's ability to easily
compose aleatoric uncertainty, epistemic uncertainty, and bias estimation
together in a single procedure, and show how this approach provides a
comprehensive awareness of NN risk.

通过扩展模型实现对风险的认知，并利用不同算法量化不同风险指标，以此提供深度神经网络的全面风险感知。

Capsa：深度神经网络风险量化的统一框架

Capsa: A Unified Framework for Quantifying Risk in Deep Neural Networks

Humans exhibit disagreement during data labeling. We term this disagreement
as human label uncertainty. In this work, we study the ramifications of human
label uncertainty (HLU). Our evaluation of existing uncertainty estimation
algorithms, with the presence of HLU, indicates the limitations of existing
uncertainty metrics and algorithms themselves in response to HLU. Meanwhile, we
observe undue effects in predictive uncertainty and generalizability. To
mitigate the undue effects, we introduce a novel natural scene statistics (NSS)
based label dilution training scheme without requiring massive human labels.
Specifically, we first select a subset of samples with low perceptual quality
ranked by statistical regularities of images. We then assign separate labels to
each sample in this subset to obtain a training set with diluted labels. Our
experiments and analysis demonstrate that training with NSS-based label
dilution alleviates the undue effects caused by HLU.

本研究探讨了人类标注不确定性对于现有的不确定性度量和算法本身的限制，引入一种基于自然场景统计的标签稀释训练方案，减轻人类标注不确定性带来的不适当影响。