Human-annotated data plays a critical role in the fairness of AI systems,
including those that deal with life-altering decisions or moderating
human-created web/social media content. Conventionally, annotator disagreements
are resolved before any learning takes place. However, researchers are
increasingly identifying annotator disagreement as pervasive and meaningful.
They also question the performance of a system when annotators disagree.
Particularly when minority views are disregarded, especially among groups that
may already be underrepresented in the annotator population. In this paper, we
introduce \emph{CrowdOpinion}\footnote{Accepted for publication at ACL 2023},
an unsupervised learning based approach that uses language features and label
distributions to pool similar items into larger samples of label distributions.
We experiment with four generative and one density-based clustering method,
applied to five linear combinations of label distributions and features. We use
five publicly available benchmark datasets (with varying levels of annotator
disagreements) from social media (Twitter, Gab, and Reddit). We also experiment
in the wild using a dataset from Facebook, where annotations come from the
platform itself by users reacting to posts. We evaluate \emph{CrowdOpinion} as
a label distribution prediction task using KL-divergence and a single-label
problem using accuracy measures.

通过使用语言特征和标签分布，我们介绍了一种无监督学习方法 CrowdOpinion，用于将类似项目汇集为更大的标签分布样本，其中涉及人工智能系统的公平性和人工注释者不一致问题的解决。