Reasoning in the real world is not divorced from situations. How to capture
the present knowledge from surrounding situations and perform reasoning
accordingly is crucial and challenging for machine intelligence. This paper
introduces a new benchmark that evaluates the situated reasoning ability via
situation abstraction and logic-grounded question answering for real-world
videos, called Situated Reasoning in Real-World Videos (STAR Benchmark). This
benchmark is built upon the real-world videos associated with human actions or
interactions, which are naturally dynamic, compositional, and logical. The
dataset includes four types of questions, including interaction, sequence,
prediction, and feasibility. We represent the situations in real-world videos
by hyper-graphs connecting extracted atomic entities and relations (e.g.,
actions, persons, objects, and relationships). Besides visual perception,
situated reasoning also requires structured situation comprehension and logical
reasoning. Questions and answers are procedurally generated. The answering
logic of each question is represented by a functional program based on a
situation hyper-graph. We compare various existing video reasoning models and
find that they all struggle on this challenging situated reasoning task. We
further propose a diagnostic neuro-symbolic model that can disentangle visual
perception, situation abstraction, language understanding, and functional
reasoning to understand the challenges of this benchmark.

通过对真实世界的视频进行情境抽象和逻辑基础问题回答，本文介绍了一种评估情境推理能力的新基准，称为 Situated Reasoning in Real-World Videos (STAR Benchmark)。该基准通过与人类动作或互动相关的真实世界视频构建，涵盖了相互作用、序列、预测和可行性等四种类型的问题。研究表明，各种现有的视频推理模型都在这一具有挑战性的情境推理任务上遇到了困难，因此我们进一步提出了一种诊断性神经符号模型，用于解决这一基准所面临的挑战。