Facing the current debate on whether Large Language Models (LLMs) attain
near-human intelligence levels (Mitchell & Krakauer, 2023; Bubeck et al., 2023;
Kosinski, 2023; Shiffrin & Mitchell, 2023; Ullman, 2023), the current study
introduces a benchmark for evaluating social intelligence, one of the most
distinctive aspects of human cognition. We developed a comprehensive
theoretical framework for social dynamics and introduced two evaluation tasks:
Inverse Reasoning (IR) and Inverse Inverse Planning (IIP). Our approach also
encompassed a computational model based on recursive Bayesian inference, adept
at elucidating diverse human behavioral patterns. Extensive experiments and
detailed analyses revealed that humans surpassed the latest GPT models in
overall performance, zero-shot learning, one-shot generalization, and
adaptability to multi-modalities. Notably, GPT models demonstrated social
intelligence only at the most basic order (order = 0), in stark contrast to
human social intelligence (order >= 2). Further examination indicated a
propensity of LLMs to rely on pattern recognition for shortcuts, casting doubt
on their possession of authentic human-level social intelligence. Our codes,
dataset, appendix and human data are released at
this https URL

该研究提出了一个用于评估社交智能的基准，引入了逆向推理和逆向规划这两个评估任务，通过基于递归贝叶斯推理的计算模型对人类行为模式进行解释，实验证明了人类在整体表现、零样本学习、一次推广和多模态适应性方面超越了最新的 GPT 模型，并显示出 LLMs 倾向于依赖模式识别来进行快速计算，对其是否拥有真正的人类级社交智能产生了质疑。