We propose SETI (Systematicity Evaluation of Textual Inference), a novel and
comprehensive benchmark designed for evaluating pre-trained language models
(PLMs) for their systematicity capabilities in the domain of textual inference.
Specifically, SETI offers three different NLI tasks and corresponding datasets
to evaluate various types of systematicity in reasoning processes. In order to
solve these tasks, models are required to perform compositional inference based
on known primitive constituents. We conduct experiments of SETI on six widely
used PLMs. Results show that various PLMs are able to solve unseen
compositional inferences when having encountered the knowledge of how to
combine primitives, with good performance. However, they are considerably
limited when this knowledge is unknown to the model (40-100% points decrease).
Furthermore, we find that PLMs can improve drastically once exposed to crucial
compositional knowledge in minimalistic shots. These findings position SETI as
the first benchmark for measuring the future progress of PLMs in achieving
systematicity generalization in the textual inference.

SETI 是一个用于评估预训练语言模型（PLMs）在文本推理领域中系统能力的新型综合性基准。它提供了三种不同的 NLI 任务和相应的数据集来评估模型在推理过程中各种类型的系统性。