Recently, large pre-trained language models (LLMs) have demonstrated superior
language understanding abilities, including zero-shot causal reasoning.
However, it is unclear to what extent their capabilities are similar to human
ones. We here study the processing of an event $B$ in a script-based story,
which causally depends on a previous event $A$. In our manipulation, event $A$
is stated, negated, or omitted in an earlier section of the text. We first
conducted a self-paced reading experiment, which showed that humans exhibit
significantly longer reading times when causal conflicts exist ($\neg A
\rightarrow B$) than under logical conditions ($A \rightarrow B$). However,
reading times remain similar when cause A is not explicitly mentioned,
indicating that humans can easily infer event B from their script knowledge. We
then tested a variety of LLMs on the same data to check to what extent the
models replicate human behavior. Our experiments show that 1) only recent LLMs,
like GPT-3 or Vicuna, correlate with human behavior in the $\neg A \rightarrow
B$ condition. 2) Despite this correlation, all models still fail to predict
that $nil \rightarrow B$ is less surprising than $\neg A \rightarrow B$,
indicating that LLMs still have difficulties integrating script knowledge. Our
code and collected data set are available at
this https URL

最近，大规模预训练语言模型（LLM）展示了卓越的语言理解能力，包括零示例因果推理。然而，目前尚不清楚它们的能力在多大程度上与人类相似。本文研究了一个基于脚本故事中事件 $B$ 的处理，该事件的因果关系依赖于前一个事件 $A$。通过自行阅读实验，发现在因果冲突存在（$
eg A ightarrow B$）的情况下，人类的阅读时间明显较长，而在逻辑条件（$A ightarrow B$）下，阅读时间保持类似。然而，当未明确提及事件 A 时，阅读时间仍然类似，表明人类可以轻易从脚本知识中推断出事件 B。随后，我们测试了多种 LLM 模型在相同数据上的表现，以检查模型在多大程度上复制了人类行为。实验结果显示：1）只有最近的 LLM 模型，如 GPT-3 或 Vicuna，与人类行为在 $
eg A ightarrow B$ 条件下相关。2）尽管存在这种相关性，所有模型仍然无法预测 $nil ightarrow B$ 相对于 $
eg A ightarrow B$ 来说更不足为奇，表明 LLM 模型在整合脚本知识方面仍存在困难。我们的代码和收集的数据集可在此 https URL 获取。