Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters' perceptions of offense. Additionally, we utilize CrowdTruth's rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.

本研究旨在解决人类反馈中普遍存在的意见分歧问题，特别是在政治敏感环境中的评审者之间的对立信仰和价值观。论文提出了一种新颖的代偿注释方法，并通过分析评审者凝聚力和质量来探讨政治倾向及人口背景对评审者感知的影响，最终为提高人类中心AI系统的评审质量提供了重要洞见。

从代偿视角看评审者的凝聚力与质量