The ability to reason about temporal and causal events from videos lies at
the core of human intelligence. Most video reasoning benchmarks, however, focus
on pattern recognition from complex visual and language input, instead of on
causal structure. We study the complementary problem, exploring the temporal
and causal structures behind videos of objects with simple visual appearance.
To this end, we introduce the CoLlision Events for Video REpresentation and
Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of
computational models on a wide range of reasoning tasks. Motivated by the
theory of human casual judgment, CLEVRER includes four types of questions:
descriptive (e.g., "what color"), explanatory ("what is responsible for"),
predictive ("what will happen next"), and counterfactual ("what if"). We
evaluate various state-of-the-art models for visual reasoning on our benchmark.
While these models thrive on the perception-based task (descriptive), they
perform poorly on the causal tasks (explanatory, predictive and
counterfactual), suggesting that a principled approach for causal reasoning
should incorporate the capability of both perceiving complex visual and
language inputs, and understanding the underlying dynamics and causal
relations. We also study an oracle model that explicitly combines these
components via symbolic representations.

介绍了一种新的视频数据集 CLEVRER，以组合描述、解释、预测和假设性问题的方式评估计算模型，结果表明现有的视觉推理模型在因果推理任务（解释性、预测性和假设性）方面表现不佳，需要在模型中融合语言输入和因果关系的理解。