Researchers have raised awareness about the harms of aggregating labels
especially in subjective tasks that naturally contain disagreements among human
annotators. In this work we show that models that are only provided aggregated
labels show low confidence on high-disagreement data instances. While previous
studies consider such instances as mislabeled, we argue that the reason the
high-disagreement text instances have been hard-to-learn is that the
conventional aggregated models underperform in extracting useful signals from
subjective tasks. Inspired by recent studies demonstrating the effectiveness of
learning from raw annotations, we investigate classifying using Multiple Ground
Truth (Multi-GT) approaches. Our experiments show an improvement of confidence
for the high-disagreement instances.

研究人员提高了对聚合标签的伤害的认识，特别是在自然存在人类标注员之间存在分歧的主观任务中。本文展示了仅仅提供聚合标签的模型在高度分歧的数据实例上表现出较低的置信度。我们认为过去的研究将这些实例视为标注错误，但我们认为高度分歧的文本实例难以学习的原因是传统的聚合模型在从主观任务中提取有用信号方面表现不佳。受最近一些研究表明从原始标注中学习的有效性的启发，我们研究了使用多个真实标准的分类方法。我们的实验显示在高度分歧实例上有置信度的提高。