Logical reasoning is central to complex human activities, such as thinking,
debating, and planning; it is also a central component of many AI systems as
well. In this paper, we investigate the extent to which encoder-only
transformer language models (LMs) can reason according to logical rules. We ask
whether those LMs can deduce theorems in propositional calculus and first-order
logic; if their relative success in these problems reflects general logical
capabilities; and which layers contribute the most to the task. First, we show
for several encoder-only LMs that they can be trained, to a reasonable degree,
to determine logical validity on various datasets. Next, by cross-probing
fine-tuned models on these datasets, we show that LMs have difficulty in
transferring their putative logical reasoning ability, which suggests that they
may have learned dataset-specific features, instead of a general capability.
Finally, we conduct a layerwise probing experiment, which shows that the
hypothesis classification task is mostly solved through higher layers.

本文研究了仅编码器变换器语言模型在逻辑规则推理方面的能力，并通过多个数据集的实验结果表明，这些语言模型在确定逻辑有效性上取得了合理的程度，但在迁移能力方面存在困难，可能是学习了特定数据集的特征而不是一般的能力，同时通过分层探测实验证明假设分类任务主要是通过较高层解决的。