A number of recent benchmarks seek to assess how well models handle natural
language negation. However, these benchmarks lack the controlled example
paradigms that would allow us to infer whether a model had learned how negation
morphemes semantically scope. To fill these analytical gaps, we present the
Scoped Negation NLI (ScoNe-NLI) benchmark, which contains contrast sets of six
examples with up to two negations where either zero, one, or both negative
morphemes affect the NLI label. We use ScoNe-NLI to assess fine-tuning and
in-context learning strategies. We find that RoBERTa and DeBERTa models solve
ScoNe-NLI after many shot fine-tuning. For in-context learning, we test
InstructGPT models and find that most prompt strategies are not successful,
including those using step-by-step reasoning. To better understand this result,
we extend ScoNe with ScoNe-NLG, a sentence completion test set that embeds
negation reasoning in short narratives. Here, InstructGPT is successful, which
reveals the model can correctly reason about negation, but struggles to do so
on prompt-adapted NLI examples outside of its core pretraining regime.

本文介绍了 Scoped Negation NLI 基准测试，用于评估自然语言模型处理否定语义组合的能力，其中用到了一系列例子和策略，最后发现 InstructGPT 模型在句子补全测试中表现出了正确的否定语义推理能力，但在处理预训练体系外的样例时表现无法胜任。