We introduce ECHo, a diagnostic dataset of event causality inference grounded
in visual-and-linguistic social scenarios. ECHo employs real-world
human-centric deductive information collected from crime drama, bridging the
gap in multimodal reasoning towards higher social intelligence through the
elicitation of intermediate Theory-of-Mind (ToM). We propose a unified
framework aligned with the Chain-of-Thought (CoT) paradigm to assess the
reasoning capability of current AI systems. This ToM-enhanced CoT pipeline can
accommodate and integrate various large foundation models in zero-shot
visual-and-linguistic understanding. With this framework, we scrutinize the
advanced large language and multimodal models via three complementary
human-centric ECHo tasks. Further analysis demonstrates ECHo as a challenging
dataset to expose imperfections and inconsistencies in reasoning.

该论文介绍了一种基于真实世界人类推断信息的诊断数据集 ECHo，旨在通过理论设计闭环链来评估当前人工智能系统的信息推理能力。该系统在零 - shot 视觉和语言理解中融合了各类大型基础模型，并进一步通过三个不同的任务来证明 ECHo 作为一个挑战性的数据集，可以揭示信息推理中的缺陷和不一致性。