Selecting an effective training signal for tasks in natural language
processing is difficult: collecting expert annotations is expensive, and
crowd-sourced annotations may not be reliable. At the same time, recent work in
machine learning has demonstrated that learning from soft-labels acquired from
crowd annotations can be effective, especially when there is distribution shift
in the test set. However, the best method for acquiring these soft labels is
inconsistent across tasks. This paper proposes new methods for acquiring
soft-labels from crowd-annotations by aggregating the distributions produced by
existing methods. In particular, we propose to find a distribution over classes
by learning from multiple-views of crowd annotations via temperature scaling
and finding the Jensen-Shannon centroid of their distributions. We demonstrate
that using these aggregation methods leads to best or near-best performance
across four NLP tasks on out-of-domain test sets, mitigating fluctuations in
performance when using the constituent methods on their own. Additionally,
these methods result in best or near-best uncertainty estimation across tasks.
We argue that aggregating different views of crowd-annotations as soft-labels
is an effective way to ensure performance which is as good or better than the
best individual view, which is useful given the inconsistency in performance of
the individual methods.

本文提出了新的方法以通过温度调节和 Jensen-Shannon 中心法从众包注释中汇总多个视角的分布，从而获得软标签。我们展示了这些聚合方法在四项自然语言处理任务中表现出最佳或接近最佳的性能，同时还验证了这些方法在任务不确定性估计上表现最佳。

基于群众标注的多视角知识蒸馏用于领域外泛化

Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization

Identifying the quality of free-text arguments has become an important task
in the rapidly expanding field of computational argumentation. In this work, we
explore the challenging task of argument quality ranking. To this end, we
created a corpus of 30,497 arguments carefully annotated for point-wise
quality, released as part of this work. To the best of our knowledge, this is
the largest dataset annotated for point-wise argument quality, larger by a
factor of five than previously released datasets. Moreover, we address the core
issue of inducing a labeled score from crowd annotations by performing a
comprehensive evaluation of different approaches to this problem. In addition,
we analyze the quality dimensions that characterize this dataset. Finally, we
present a neural method for argument quality ranking, which outperforms several
baselines on our own dataset, as well as previous methods published for another
dataset.

本研究探索了论证质量排序的挑战性任务，构建了一个被精心注释的语料库来解决这个问题，并进行了综合评估，最后提出了一种神经方法来解决此问题。