Human label variation, or annotation disagreement, exists in many natural
language processing (NLP) tasks, including natural language inference (NLI). To
gain direct evidence of how NLI label variation arises, we build LiveNLI, an
English dataset of 1,415 ecologically valid explanations (annotators explain
the NLI labels they chose) for 122 MNLI items (at least 10 explanations per
item). The LiveNLI explanations confirm that people can systematically vary on
their interpretation and highlight within-label variation: annotators sometimes
choose the same label for different reasons. This suggests that explanations
are crucial for navigating label interpretations in general. We few-shot prompt
large language models to generate explanations but the results are
inconsistent: they sometimes produces valid and informative explanations, but
it also generates implausible ones that do not support the label, highlighting
directions for improvement.

人类标签差异存在于许多自然语言处理任务中，包括自然语言推理。在研究中，我们建立了 LiveNLI 数据集，通过 1415 个生态有效解释（标注者解释他们选择的 NLI 标签）来获取直接证据，研究了标签差异的产生。结果显示，解释对于理解标签的不同解释至关重要，并且标注者有时会基于不同的原因选择相同的标签，这表明解释在一般中导航标签解释方面起着关键作用。我们进行了少样本大语言模型生成解释的尝试，但结果不一致：它有时会产生有效和有信息量的解释，但也会生成不支持标签的不合理的解释，提出了改进的方向。

自然有效的自然语言推理标签变异解释

Ecologically Valid Explanations for Label Variation in NLI

Human label variation (Plank 2022), or annotation disagreement, exists in
many natural language processing (NLP) tasks. To be robust and trusted, NLP
models need to identify such variation and be able to explain it. To this end,
we created the first ecologically valid explanation dataset with diverse
reasoning, LiveNLI. LiveNLI contains annotators' highlights and free-text
explanations for the label(s) of their choice for 122 English Natural Language
Inference items, each with at least 10 annotations. We used its explanations
for chain-of-thought prompting, and found there is still room for improvement
in GPT-3's ability to predict label distribution with in-context learning.

该论文创建了一个包含注解高亮和自由文本解释的生态有效的自然语言推断数据集 --LiveNLI。使用该数据集对思维链进行提示，发现目前 GPT-3 在预测标签分布方面仍有改进的空间