Natural Language Inference (NLI) has been a cornerstone task in evaluating
language models' inferential reasoning capabilities. However, the standard
three-way classification scheme used in NLI has well-known shortcomings in
evaluating models' ability to capture the nuances of natural human reasoning.
In this paper, we argue that the operationalization of the neutral label in
current NLI datasets has low validity, is interpreted inconsistently, and that
at least one important sense of neutrality is often ignored. We uncover the
detrimental impact of these shortcomings, which in some cases leads to
annotation datasets that actually decrease performance on downstream tasks. We
compare approaches of handling annotator disagreement and identify flaws in a
recent NLI dataset that designs an annotator study based on a problematic
operationalization. Our findings highlight the need for a more refined
evaluation framework for NLI, and we hope to spark further discussion and
action in the NLP community.

本文指出自然语言推理领域在评估模型推理推理能力方面所使用的标准三分法存在着缺陷，尤其是与人类推理过程的案例感知能力的差异。作者还发现现有的自然语言推理标注数据集中的一种中性标签处于低效的状态，且存在解释不一致的情况，而至少一种重要的得分中性标签经常被忽略。我们比较了处理不同标注者之间的分歧的方法，并确定了最近的自然语言推理（NLI）数据集中设计了一个基于有问题的形式操作的注释者研究的缺陷。本文的发现强调了需要更加精确评估框架来评估 NLI，我们希望引发 NLP 社区的进一步讨论和行动。