Current pre-trained language models have enabled remarkable improvements in
downstream tasks, but it remains difficult to distinguish effects of
statistical correlation from more systematic logical reasoning grounded on the
understanding of real world. We tease these factors apart by leveraging
counterfactual conditionals, which force language models to predict unusual
consequences based on hypothetical propositions. We introduce a set of tests
from psycholinguistic experiments, as well as larger-scale controlled datasets,
to probe counterfactual predictions from five pre-trained language models. We
find that models are consistently able to override real-world knowledge in
counterfactual scenarios, and that this effect is more robust in case of
stronger baseline world knowledge -- however, we also find that for most models
this effect appears largely to be driven by simple lexical cues. When we
mitigate effects of both world knowledge and lexical cues to test knowledge of
linguistic nuances of counterfactuals, we find that only GPT-3 shows
sensitivity to these nuances, though this sensitivity is also non-trivially
impacted by lexical associative factors.

通过调查五种预训练语言模型在特殊条件下进行的逆向推理，发现这种逆向推理主要受到基于单词关联的影响，只有 GPT-3 能够展现对语言细微差别的敏感性。

反事实推理：测试语言模型对假设场景的理解

Counterfactual reasoning: Testing language models' understanding of  hypothetical scenarios

Current pre-trained language models have enabled remarkable improvements in
downstream tasks, but it remains difficult to distinguish effects of
statistical correlation from more systematic logical reasoning grounded on
understanding of the real world. In this paper we tease these factors apart by
leveraging counterfactual conditionals, which force language models to predict
unusual consequences based on hypothetical propositions. We introduce a set of
tests drawn from psycholinguistic experiments, as well as larger-scale
controlled datasets, to probe counterfactual predictions from a variety of
popular pre-trained language models. We find that models are consistently able
to override real-world knowledge in counterfactual scenarios, and that this
effect is more robust in case of stronger baseline world knowledge -- however,
we also find that for most models this effect appears largely to be driven by
simple lexical cues. When we mitigate effects of both world knowledge and
lexical cues to test knowledge of linguistic nuances of counterfactuals, we
find that only GPT-3 shows sensitivity to these nuances, though this
sensitivity is also non-trivially impacted by lexical associative factors.

本文通过使用反事实条件句，利用心理语言学实验和更大规模的数据集，对比了多种流行的预训练语言模型中的反事实预测，并发现大多数模型主要受简单的词汇线索驱动。当控制世界知识和词汇线索效应时，只有 GPT-3 在反事实细微差别的语言基础知识上显示出敏感性。