Theory of Mind (ToM) reasoning entails recognizing that other individuals
possess their own intentions, emotions, and thoughts, which is vital for
guiding one's own thought processes. Although large language models (LLMs)
excel in tasks such as summarization, question answering, and translation, they
still face challenges with ToM reasoning, especially in open-ended questions.
Despite advancements, the extent to which LLMs truly understand ToM reasoning
and how closely it aligns with human ToM reasoning remains inadequately
explored in open-ended scenarios. Motivated by this gap, we assess the
abilities of LLMs to perceive and integrate human intentions and emotions into
their ToM reasoning processes within open-ended questions. Our study utilizes
posts from Reddit's ChangeMyView platform, which demands nuanced social
reasoning to craft persuasive responses. Our analysis, comparing semantic
similarity and lexical overlap metrics between responses generated by humans
and LLMs, reveals clear disparities in ToM reasoning capabilities in open-ended
questions, with even the most advanced models showing notable limitations. To
enhance LLM capabilities, we implement a prompt tuning method that incorporates
human intentions and emotions, resulting in improvements in ToM reasoning
performance. However, despite these improvements, the enhancement still falls
short of fully achieving human-like reasoning. This research highlights the
deficiencies in LLMs' social reasoning and demonstrates how integrating human
intentions and emotions can boost their effectiveness.

利用 Reddit 的 ChangeMyView 平台的帖子，该研究评估了大型语言模型在处理心理理论推理时的能力，并通过提供人类意图和情感来增强模型表现，揭示了模型在开放式问题中与人类心理推理相比存在的差距。

LLM 是否具备类人推理能力？评估 LLM 对开放性回答的心智理论

Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs  for Open-Ended Responses

From the early days of computing, games have been important testbeds for
studying how well machines can do sophisticated decision making. In recent
years, machine learning has made dramatic advances with artificial agents
reaching superhuman performance in challenge domains like Go, Atari, and some
variants of poker. As with their predecessors of chess, checkers, and
backgammon, these game domains have driven research by providing sophisticated
yet well-defined challenges for artificial intelligence practitioners. We
continue this tradition by proposing the game of Hanabi as a new challenge
domain with novel problems that arise from its combination of purely
cooperative gameplay with two to five players and imperfect information. In
particular, we argue that Hanabi elevates reasoning about the beliefs and
intentions of other agents to the foreground. We believe developing novel
techniques for such theory of mind reasoning will not only be crucial for
success in Hanabi, but also in broader collaborative efforts, especially those
with human partners. To facilitate future research, we introduce the
open-source Hanabi Learning Environment, propose an experimental framework for
the research community to evaluate algorithmic advances, and assess the
performance of current state-of-the-art techniques.

提出将 Hanabi 游戏作为新的挑战领域来驱动人工智能研究，强调需要发展针对其他代理者信念和意图的理论推理新技术，并介绍了开源的 Hanabi 游戏学习环境以及当前最先进技术的性能评估实验框架。