Large Language Models (LLMs) have generated considerable interest and debate
regarding their potential emergence of Theory of Mind (ToM). Several recent
inquiries reveal a lack of robust ToM in these models and pose a pressing
demand to develop new benchmarks, as current ones primarily focus on different
aspects of ToM and are prone to shortcuts and data leakage. In this position
paper, we seek to answer two road-blocking questions: (1) How can we taxonomize
a holistic landscape of machine ToM? (2) What is a more effective evaluation
protocol for machine ToM? Following psychological studies, we taxonomize
machine ToM into 7 mental state categories and delineate existing benchmarks to
identify under-explored aspects of ToM. We argue for a holistic and situated
evaluation of ToM to break ToM into individual components and treat LLMs as an
agent who is physically situated in environments and socially situated in
interactions with humans. Such situated evaluation provides a more
comprehensive assessment of mental states and potentially mitigates the risk of
shortcuts and data leakage. We further present a pilot study in a grid world
setup as a proof of concept. We hope this position paper can facilitate future
research to integrate ToM with LLMs and offer an intuitive means for
researchers to better position their work in the landscape of ToM. Project
page: this https URL

大型语言模型（LLMs）在理论心理学的出现上引起了极大的兴趣和争议。我们通过心理学研究将机器的心理模拟（machine ToM）分为七个类别，并提出了一种更全面的评估方法，通过将 LLMs 放置在具体环境中进行评估，来避免捷径和数据泄露的风险。希望这篇观点论文能促进未来研究将机器 ToM 与 LLMs 结合起来，并为研究人员更好地定位自己的工作在 ToM 领域中提供一个直观的方式。