Learning commonsense reasoning from visual contexts and scenes in real-world
is a crucial step toward advanced artificial intelligence. However, existing
video reasoning benchmarks are still inadequate since they were mainly designed
for factual or situated reasoning and rarely involve broader knowledge in the
real world. Our work aims to delve deeper into reasoning evaluations,
specifically within dynamic, open-world, and structured context knowledge. We
propose a new benchmark (SOK-Bench), consisting of 44K questions and 10K
situations with instance-level annotations depicted in the videos. The
reasoning process is required to understand and apply situated knowledge and
general knowledge for problem-solving. To create such a dataset, we propose an
automatic and scalable generation method to generate question-answer pairs,
knowledge graphs, and rationales by instructing the combinations of LLMs and
MLLMs. Concretely, we first extract observable situated entities, relations,
and processes from videos for situated knowledge and then extend to open-world
knowledge beyond the visible content. The task generation is facilitated
through multiple dialogues as iterations and subsequently corrected and refined
by our designed self-promptings and demonstrations. With a corpus of both
explicit situated facts and implicit commonsense, we generate associated
question-answer pairs and reasoning processes, finally followed by manual
reviews for quality assurance. We evaluated recent mainstream large
vision-language models on the benchmark and found several insightful
conclusions. For more information, please refer to our benchmark at
www.bobbywu.com/SOKBench.

通过从真实世界的视觉环境和场景中学习常识推理，可以迈向先进的人工智能；然而，现有的视频推理基准仍然不足，因为它们主要设计用于事实推理或定位推理，很少涉及到现实世界的更广泛的知识。我们的工作旨在深入研究推理评估，尤其是在动态的、开放的和结构化的上下文知识中；我们提出了一个新的基准测试（SOK-Bench），包含 44K 个问题和 10K 个以视频为示例的情境，带有实例级别的注释。推理过程要求理解和应用情境知识和一般知识进行问题解决。为了创建这样的数据集，我们提出了一种自动且可扩展的生成方法，通过指导 LLMs 和 MLLMs 的组合来生成问题 - 答案对、知识图谱和合理性。具体来说，我们首先从视频中提取可观察到的情境实体、关系和过程，用于情境知识，然后扩展到超出可见内容的开放世界知识。通过多次对话进行任务生成，并通过我们设计的自我提示和演示进行纠正和改进。有了明确的情境事实和隐含的常识知识，我们生成相关的问题 - 答案对和推理过程，并最后通过人工审查进行质量保证。我们评估了最新的主流大视觉语言模型在基准测试上，并得出了一些有见地的结论。详情请参阅我们在 www.bobbywu.com/SOKBench 上的基准测试。