Humans make numerous inferences in text comprehension to understand
discourse. This paper aims to understand the commonalities and disparities in
the inference judgments between humans and state-of-the-art Large Language
Models (LLMs). Leveraging a comprehensively curated entailment verification
benchmark, we evaluate both human and LLM performance across various reasoning
categories. Our benchmark includes datasets from three categories (NLI,
contextual QA, and rationales) that include multi-sentence premises and
different knowledge types, thereby evaluating the inference capabilities in
complex reasoning instances. Notably, our findings reveal LLMs' superiority in
multi-hop reasoning across extended contexts, while humans excel in tasks
necessitating simple deductive reasoning. Leveraging these insights, we
introduce a fine-tuned Flan-T5 model that outperforms GPT-3.5 and rivals with
GPT-4, offering a robust open-source solution for entailment verification. As a
practical application, we showcase the efficacy of our finetuned model in
enhancing self-consistency in model-generated explanations, resulting in a 6%
performance boost on average across three multiple-choice question-answering
datasets.

该研究通过比较人类和大型语言模型（LLMs）之间的推理判断的共性和差异，以及对三个类别（NLI、contextual QA 和 rationales）的多个数据集进行评估，揭示了 LLMs 在复杂推理环境中的多步推理上的优势和人类在简单推理方面的优势，并引入了一种经过精细调整的 Flan-T5 模型来提高自洽性，在三个多项选择问答数据集上平均提高了 6% 的性能。

人与机：重新思考自然语言模型的蕴涵验证

Minds versus Machines: Rethinking Entailment Verification with Language  Models

Pre-trained Transformer-based neural architectures have consistently achieved
state-of-the-art performance in the Natural Language Inference (NLI) task.
Since NLI examples encompass a variety of linguistic, logical, and reasoning
phenomena, it remains unclear as to which specific concepts are learnt by the
trained systems and where they can achieve strong generalization. To
investigate this question, we propose a taxonomic hierarchy of categories that
are relevant for the NLI task. We introduce TAXINLI, a new dataset, that has
10k examples from the MNLI dataset (Williams et al., 2018) with these taxonomic
labels. Through various experiments on TAXINLI, we observe that whereas for
certain taxonomic categories SOTA neural models have achieved near perfect
accuracies - a large jump over the previous models - some categories still
remain difficult. Our work adds to the growing body of literature that shows
the gaps in the current NLI systems and datasets through a systematic
presentation and analysis of reasoning categories.

本文通过提出一个有关自然语言文本蕴含任务中分类的分类法以及构建一个带有该分类法的数据集 TAXINLI，研究预训练模型在不同类型蕴含问题上的表现，并发现了当前模型和数据集的不足之处。