Pre-trained Transformer-based neural architectures have consistently achieved
state-of-the-art performance in the Natural Language Inference (NLI) task.
Since NLI examples encompass a variety of linguistic, logical, and reasoning
phenomena, it remains unclear as to which specific concepts are learnt by the
trained systems and where they can achieve strong generalization. To
investigate this question, we propose a taxonomic hierarchy of categories that
are relevant for the NLI task. We introduce TAXINLI, a new dataset, that has
10k examples from the MNLI dataset (Williams et al., 2018) with these taxonomic
labels. Through various experiments on TAXINLI, we observe that whereas for
certain taxonomic categories SOTA neural models have achieved near perfect
accuracies - a large jump over the previous models - some categories still
remain difficult. Our work adds to the growing body of literature that shows
the gaps in the current NLI systems and datasets through a systematic
presentation and analysis of reasoning categories.

本文通过提出一个有关自然语言文本蕴含任务中分类的分类法以及构建一个带有该分类法的数据集 TAXINLI，研究预训练模型在不同类型蕴含问题上的表现，并发现了当前模型和数据集的不足之处。