Theory of Mind (ToM), the capacity to comprehend the mental states of
distinct individuals, is essential for numerous practical applications. With
the development of large language models, there is a heated debate about
whether they are able to perform ToM tasks. Previous studies have used
different tasks and prompts to test the ToM on large language models and the
results are inconsistent: some studies asserted these models are capable of
exhibiting ToM, while others suggest the opposite. In this study, We present
ToMChallenges, a dataset for comprehensively evaluating Theory of Mind based on
Sally-Anne and Smarties tests. We created 30 variations of each test (e.g.,
changing the person's name, location, and items). For each variation, we test
the model's understanding of different aspects: reality, belief, 1st order
belief, and 2nd order belief. We adapt our data for various tasks by creating
unique prompts tailored for each task category: Fill-in-the-Blank, Multiple
Choice, True/False, Chain-of-Thought True/False, Question Answering, and Text
Completion. If the model has a robust ToM, it should be able to achieve good
performance for different prompts across different tests. We evaluated two
GPT-3.5 models, text-davinci-003 and gpt-3.5-turbo-0301, with our datasets. Our
results indicate that consistent performance in ToM tasks remains a challenge.

本研究通过创建 ToMChallenges 数据集以综合评估 Sally-Anne 和 Smarties 测试来测试大型语言模型对理解他人的心理状态的能力，包括现实性、信念、一阶信念和二阶信念，并通过创建适用于每个任务类别的独特提示来为不同任务适应我们的数据，旨在验证模型的 ToM。我们评估了两个 GPT-3.5 模型，text-davinci-003 和 gpt-3.5-turbo-0301，结果表明，在 ToM 任务中保持一致的表现仍然是一个挑战。