Theory of Mind (ToM), the ability to understand people's minds, is an
essential ingredient for developing machines with human-level social
intelligence. Recent machine learning models, particularly large language
models, seem to show some aspects of ToM understanding. However, existing ToM
benchmarks use unimodal datasets - either video or text. Human ToM, on the
other hand, is more than video or text understanding. People can flexibly
reason about another person's mind based on conceptual representations (e.g.,
goals, beliefs, plans) extracted from any available data, which can include
visual cues, linguistic narratives, or both. To address this, we introduce a
multimodal Theory of Mind question answering (MMToM-QA) benchmark. MMToM-QA
comprehensively evaluates machine ToM both on multimodal data and on different
kinds of unimodal data about a person's activity in a household environment. To
engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian
Inverse Planning Accelerated by Language Models). BIP-ALM extracts unified
representations from multimodal data and utilizes language models for scalable
Bayesian inverse planning. We conducted a systematic comparison of human
performance, BIP-ALM, and state-of-the-art models, including GPT-4. The
experiments demonstrate that large language models and large multimodal models
still lack robust ToM capacity. BIP-ALM, on the other hand, shows promising
results, by leveraging the power of both model-based mental inference and
language models.

为了发展具有人类级社交智能的机器，我们引入了一种多模式心智问答（MMToM-QA）基准，它综合评估机器的心智理解能力，并基于多模式数据和关于家庭环境中人物活动的不同类别的单模式数据。为了实现多模式心智能力，提出了一种新方法 BIP-ALM（基于语言模型的贝叶斯逆规划加速），它从多模式数据中提取统一的表示，并利用语言模型进行可扩展的贝叶斯逆规划。与人类表现、BIP-ALM 和包括 GPT-4 在内的先进模型进行了系统比较，实验表明大型语言模型和大型多模式模型仍然缺乏强大的心智理解能力，而 BIP-ALM 则通过利用基于模型的心智推理和语言模型的能力展现了有希望的结果。