Entities and events are crucial to natural language reasoning and common in
procedural texts. Existing work has focused either exclusively on entity state
tracking (e.g., whether a pan is hot) or on event reasoning (e.g., whether one
would burn themselves by touching the pan), while these two tasks are often
causally related. We propose CREPE, the first benchmark on causal reasoning of
event plausibility and entity states. We show that most language models,
including GPT-3, perform close to chance at .35 F1, lagging far behind human at
.87 F1. We boost model performance to .59 F1 by creatively representing events
as programming languages while prompting language models pretrained on code. By
injecting the causal relations between entities and events as intermediate
reasoning steps in our representation, we further boost the performance to .67
F1. Our findings indicate not only the challenge that CREPE brings for language
models, but also the efficacy of code-like prompting combined with
chain-of-thought prompting for multihop event reasoning.

本研究提出 CREPE 作为第一个事件因果推理和实体状态的基准测试，并在其中发现大多数语言模型表现低迷，但通过将事件表示为编程语言，并在表示中注入实体和事件之间的因果关系，提高了模型的性能。