Can large multimodal models have a human-like ability for emotional and
social reasoning, and if so, how does it work? Recent research has discovered
emergent theory-of-mind (ToM) reasoning capabilities in large language models
(LLMs). LLMs can reason about people's mental states by solving various
text-based ToM tasks that ask questions about the actors' ToM (e.g., human
belief, desire, intention). However, human reasoning in the wild is often
grounded in dynamic scenes across time. Thus, we consider videos a new medium
for examining spatio-temporal ToM reasoning ability. Specifically, we ask
explicit probing questions about videos with abundant social and emotional
reasoning content. We develop a pipeline for multimodal LLM for ToM reasoning
using video and text. We also enable explicit ToM reasoning by retrieving key
frames for answering a ToM question, which reveals how multimodal LLMs reason
about ToM.

大型多模态模型能否拥有类似人类的情感和社交推理能力，以及这是如何实现的？近期研究已经发现了大型语言模型的浮现理论推理能力。通过解决各种基于文本的涉及认知理论（如人类信念、欲望和意图）的任务，语言模型能够推理人们的心理状态。然而，人类的推理能力通常基于动态场景和时间。因此，我们认为视频是一个检验时空认知推理能力的新媒介。我们通过使用视频和文本开发了一个基于多模态语言模型的推理管道，实现了对涉及社交和情感推理内容的视频的显性推理。同时，我们还通过检索回答推理问题所需的关键帧来启用显性认知推理，揭示了多模态语言模型如何推理认知理论。

通过心灵之眼理论：用多模态视频和大型语言模型阅读思维

Through the Theory of Mind's Eye: Reading Minds with Multimodal Video  Large Language Models

Theory of Mind (ToM) reasoning entails recognizing that other individuals
possess their own intentions, emotions, and thoughts, which is vital for
guiding one's own thought processes. Although large language models (LLMs)
excel in tasks such as summarization, question answering, and translation, they
still face challenges with ToM reasoning, especially in open-ended questions.
Despite advancements, the extent to which LLMs truly understand ToM reasoning
and how closely it aligns with human ToM reasoning remains inadequately
explored in open-ended scenarios. Motivated by this gap, we assess the
abilities of LLMs to perceive and integrate human intentions and emotions into
their ToM reasoning processes within open-ended questions. Our study utilizes
posts from Reddit's ChangeMyView platform, which demands nuanced social
reasoning to craft persuasive responses. Our analysis, comparing semantic
similarity and lexical overlap metrics between responses generated by humans
and LLMs, reveals clear disparities in ToM reasoning capabilities in open-ended
questions, with even the most advanced models showing notable limitations. To
enhance LLM capabilities, we implement a prompt tuning method that incorporates
human intentions and emotions, resulting in improvements in ToM reasoning
performance. However, despite these improvements, the enhancement still falls
short of fully achieving human-like reasoning. This research highlights the
deficiencies in LLMs' social reasoning and demonstrates how integrating human
intentions and emotions can boost their effectiveness.

利用 Reddit 的 ChangeMyView 平台的帖子，该研究评估了大型语言模型在处理心理理论推理时的能力，并通过提供人类意图和情感来增强模型表现，揭示了模型在开放式问题中与人类心理推理相比存在的差距。

LLM 是否具备类人推理能力？评估 LLM 对开放性回答的心智理论

Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs  for Open-Ended Responses

Large Language Models (LLMs) have recently shown a promise and emergence of
Theory of Mind (ToM) ability and even outperform humans in certain ToM tasks.
To evaluate and extend the boundaries of the ToM reasoning ability of LLMs, we
propose a novel concept, taxonomy, and framework, the ToM reasoning with Zero,
Finite, and Infinite Belief History and develop a multi-round text-based game,
called $\textit{Pick the Right Stuff}$, as a benchmark. We have evaluated six
LLMs with this game and found their performance on Zero Belief History is
consistently better than on Finite Belief History. In addition, we have found
two of the models with small parameter sizes outperform all the evaluated
models with large parameter sizes. We expect this work to pave the way for
future ToM benchmark development and also for the promotion and development of
more complex AI agents or systems which are required to be equipped with more
complex ToM reasoning ability.

在这篇论文中，我们提出了一种新的概念、分类法和框架 —— 使用零、有限和无限信仰历史进行理论心智推理，并开发了一个名为 'Pick the Right Stuff' 的多回合文本游戏作为一个基准。我们评估了六个大型语言模型在这个游戏中的表现，并发现它们在零信仰历史上的性能一致优于有限信仰历史。此外，我们还发现其中两个参数规模较小的模型胜过了所有参数规模较大的模型。我们期望这项工作为未来的理论心智基准开发铺平道路，也为需要更复杂的理论心智推理能力的更复杂的人工智能代理或系统的推进和发展提供支持。