Negation is poorly captured by current language models, although the extent
of this problem is not widely understood. We introduce a natural language
inference (NLI) test suite to enable probing the capabilities of NLP methods,
with the aim of understanding sub-clausal negation. The test suite contains
premise--hypothesis pairs where the premise contains sub-clausal negation and
the hypothesis is constructed by making minimal modifications to the premise in
order to reflect different possible interpretations. Aside from adopting
standard NLI labels, our test suite is systematically constructed under a
rigorous linguistic framework. It includes annotation of negation types and
constructions grounded in linguistic theory, as well as the operations used to
construct hypotheses. This facilitates fine-grained analysis of model
performance. We conduct experiments using pre-trained language models to
demonstrate that our test suite is more challenging than existing benchmarks
focused on negation, and show how our annotation supports a deeper
understanding of the current NLI capabilities in terms of negation and
quantification.

本文介绍了一种自然语言推理测试套件，以探索自然语言处理方法的能力，理解子句否定。该测试套件包含基于严格的语言学框架进行的否定类型和构造的注释，以及用于构造假设的操作。作者使用预训练语言模型进行实验，证明了此测试套件比现有的侧重于否定的基准更具挑战性，并展示了作者的注释如何支持对当前自然语言推理能力在否定和量化方面的更深入理解。