Neural Theory-of-Mind (N-ToM), machine's ability to understand and keep track
of the mental states of others, is pivotal in developing socially intelligent
agents. However, prevalent N-ToM benchmarks have several shortcomings,
including the presence of ambiguous and artificial narratives, absence of
personality traits and preferences, a lack of questions addressing characters'
psychological mental states, and limited diversity in the questions posed. In
response to these issues, we construct OpenToM, a new benchmark for assessing
N-ToM with (1) longer and clearer narrative stories, (2) characters with
explicit personality traits, (3) actions that are triggered by character
intentions, and (4) questions designed to challenge LLMs' capabilities of
modeling characters' mental states of both the physical and psychological
world. Using OpenToM, we reveal that state-of-the-art LLMs thrive at modeling
certain aspects of mental states in the physical world but fall short when
tracking characters' mental states in the psychological world.

机器学习中的神经心智理论 (N-ToM) 是理解和跟踪他人心理状态的关键，本研究构建了一个新的基准 OpenToM，通过长而清晰的叙述、具有明确人格特征的角色以及挑战人工智能模型对心理和心理世界中角色心理状态建模能力的设计提出了一些问题，揭示了最先进的模型在物理世界中对心理状态的建模方面表现出色，但在心理世界中的心理状态跟踪方面表现不足。

OpenToM: 评估大型语言模型理解他人心理思维能力的全面基准

OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind  Reasoning Capabilities of Large Language Models

The escalating debate on AI's capabilities warrants developing reliable
metrics to assess machine "intelligence". Recently, many anecdotal examples
were used to suggest that newer large language models (LLMs) like ChatGPT and
GPT-4 exhibit Neural Theory-of-Mind (N-ToM); however, prior work reached
conflicting conclusions regarding those abilities. We investigate the extent of
LLMs' N-ToM through an extensive evaluation on 6 tasks and find that while LLMs
exhibit certain N-ToM abilities, this behavior is far from being robust. We
further examine the factors impacting performance on N-ToM tasks and discover
that LLMs struggle with adversarial examples, indicating reliance on shallow
heuristics rather than robust ToM abilities. We caution against drawing
conclusions from anecdotal examples, limited benchmark testing, and using
human-designed psychological tests to evaluate models.

我们通过在 6 个任务上进行广泛的评估，发现虽然 LLM 表现出某些神经理论社交认知能力，但这种行为远非稳健。我们进一步研究影响 N-ToM 任务表现的因素，并发现 LLM 难以应对对抗性例子，这表明它们依赖于浅层启发式算法，而非稳健的 ToM 能力。我们警告不要从个别例子、有限的基准测试和使用人设计的心理测试来评估模型。