Existing Natural Language Inference (NLI) datasets, while being instrumental
in the advancement of Natural Language Understanding (NLU) research, are not
related to scientific text. In this paper, we introduce SciNLI, a large dataset
for NLI that captures the formality in scientific text and contains 107,412
sentence pairs extracted from scholarly papers on NLP and computational
linguistics. Given that the text used in scientific literature differs vastly
from the text used in everyday language both in terms of vocabulary and
sentence structure, our dataset is well suited to serve as a benchmark for the
evaluation of scientific NLU models. Our experiments show that SciNLI is harder
to classify than the existing NLI datasets. Our best performing model with
XLNet achieves a Macro F1 score of only 78.18% and an accuracy of 78.23%
showing that there is substantial room for improvement.

本文介绍了 SciNLI，它是一个用于 NLU 的大型 NLI 数据集，旨在捕捉科学文本中的规范性，并包含 107,412 个从 NLP 和计算语言学方面的学术论文中提取的句子对。我们的实验表明，SciNLI 比现有的 NLI 数据集更难分类。使用 XLNet 的最佳模型仅实现了 78.18％的 Macro F1 得分和 78.23％的准确度，表明还有很大的改进空间。