Entities and events are crucial to natural language reasoning and common in
procedural texts. Existing work has focused either exclusively on entity state
tracking (e.g., whether a pan is hot) or on event reasoning (e.g., whether one
would burn themselves by touching the pan), while these two tasks are often
causally related. We propose CREPE, the first benchmark on causal reasoning of
event plausibility and entity states. We show that most language models,
including GPT-3, perform close to chance at .35 F1, lagging far behind human at
.87 F1. We boost model performance to .59 F1 by creatively representing events
as programming languages while prompting language models pretrained on code. By
injecting the causal relations between entities and events as intermediate
reasoning steps in our representation, we further boost the performance to .67
F1. Our findings indicate not only the challenge that CREPE brings for language
models, but also the efficacy of code-like prompting combined with
chain-of-thought prompting for multihop event reasoning.

本研究提出 CREPE 作为第一个事件因果推理和实体状态的基准测试，并在其中发现大多数语言模型表现低迷，但通过将事件表示为编程语言，并在表示中注入实体和事件之间的因果关系，提高了模型的性能。

程序性文本中实体和事件的因果推理

Causal Reasoning of Entities and Events in Procedural Texts

Information seeking users often pose questions with false presuppositions,
especially when asking about unfamiliar topics. Most existing question
answering (QA) datasets, in contrast, assume all questions have well defined
answers. We introduce CREPE, a QA dataset containing a natural distribution of
presupposition failures from online information-seeking forums. We find that
25% of questions contain false presuppositions, and provide annotations for
these presuppositions and their corrections. Through extensive baseline
experiments, we show that adaptations of existing open-domain QA models can
find presuppositions moderately well, but struggle when predicting whether a
presupposition is factually correct. This is in large part due to difficulty in
retrieving relevant evidence passages from a large text corpus. CREPE provides
a benchmark to study question answering in the wild, and our analyses provide
avenues for future work in better modeling and further studying the task.

介绍了一个包含在线信息获取论坛上的自然分布条件失败的问答数据集 CREPE，其中 25％的问题包含虚假前提；通过实验表明现有的开放域问答模型可以适当地找到前提，但很难预测前提是否属实，CREPE 提供了研究野外问答的基准，并为更好地建模和深入研究任务提供了途径。