The NLP community has long advocated for the construction of multi-annotator
datasets to better capture the nuances of language interpretation,
subjectivity, and ambiguity. This paper conducts a retrospective study to show
how performance scores can vary when a dataset expands from a single annotation
per instance to multiple annotations. We propose a novel multi-annotator
simulation process to generate datasets with varying annotation budgets. We
show that similar datasets with the same annotation budget can lead to varying
performance gains. Our findings challenge the popular belief that models
trained on multi-annotation examples always lead to better performance than
models trained on single or few-annotation examples.

多次注释数据集的性能分数可能会因数据集扩大，从单个注释变为多个注释而有所不同，我们的研究挑战了传统观念，表明使用多次注释的模型并不总能比使用少次或单次注释的模型具有更好的性能。

揭示多注释过程：以注释数量和样本难度对模型性能的影响为例

Unveiling the Multi-Annotation Process: Examining the Influence of  Annotation Quantity and Instance Difficulty on Model Performance

Despite widespread use of LLMs as conversational agents, evaluations of
performance fail to capture a crucial aspect of communication: interpreting
language in context. Humans interpret language using beliefs and prior
knowledge about the world. For example, we intuitively understand the response
"I wore gloves" to the question "Did you leave fingerprints?" as meaning "No".
To investigate whether LLMs have the ability to make this type of inference,
known as an implicature, we design a simple task and evaluate widely used
state-of-the-art models. We find that, despite only evaluating on utterances
that require a binary inference (yes or no), most perform close to random.
Models adapted to be "aligned with human intent" perform much better, but still
show a significant gap with human performance. We present our findings as the
starting point for further research into evaluating how LLMs interpret language
in context and to drive the development of more pragmatic and useful models of
human discourse.

这篇论文研究 LLMs 在理解语境方面的能力，通过实验发现目前的 LLMs 模型在完成只需要二进制推断的对话时表现平庸，需要进一步研究如何使得 LLMs 适应更贴合人类意图的对话模式。