Building machines that can reason about physical events and their causal
relationships is crucial for flexible interaction with the physical world.
However, most existing physical and causal reasoning benchmarks are exclusively
based on synthetically generated events and synthetic natural language
descriptions of causal relationships. This design brings up two issues. First,
there is a lack of diversity in both event types and natural language
descriptions; second, causal relationships based on manually-defined heuristics
are different from human judgments. To address both shortcomings, we present
the CLEVRER-Humans benchmark, a video reasoning dataset for causal judgment of
physical events with human labels. We employ two techniques to improve data
collection efficiency: first, a novel iterative event cloze task to elicit a
new representation of events in videos, which we term Causal Event Graphs
(CEGs); second, a data augmentation technique based on neural language
generative models. We convert the collected CEGs into questions and answers to
be consistent with prior work. Finally, we study a collection of baseline
approaches for CLEVRER-Humans question-answering, highlighting the great
challenges set forth by our benchmark.

构建能推理物理事件及其因果关系的机器对于与物质世界进行灵活交互至关重要。为了解决现有物理和因果推理基准的缺点，我们提出了一种视频推理数据集，以人类标签为基础，用于对物理事件的因果判断。