Social intelligence and Theory of Mind (ToM), i.e., the ability to reason
about the different mental states, intents, and reactions of all people
involved, allow humans to effectively navigate and understand everyday social
interactions. As NLP systems are used in increasingly complex social
situations, their ability to grasp social dynamics becomes crucial.
In this work, we examine the open question of social intelligence and Theory
of Mind in modern NLP systems from an empirical and theory-based perspective.
We show that one of today's largest language models (GPT-3; Brown et al., 2020)
lacks this kind of social intelligence out-of-the box, using two tasks:
SocialIQa (Sap et al., 2019), which measures models' ability to understand
intents and reactions of participants of social interactions, and ToMi (Le et
al., 2019), which measures whether models can infer mental states and realities
of participants of situations.
Our results show that models struggle substantially at these Theory of Mind
tasks, with well-below-human accuracies of 55% and 60% on SocialIQa and ToMi,
respectively. To conclude, we draw on theories from pragmatics to contextualize
this shortcoming of large language models, by examining the limitations
stemming from their data, neural architecture, and training paradigms.
Challenging the prevalent narrative that only scale is needed, we posit that
person-centric NLP approaches might be more effective towards neural Theory of
Mind.

本文从经验和理论角度出发，探究了现代 NLP 系统中的社交智能和心理理论的问题，并使用 SocialIQa 和 ToMi 任务检验了 GPT-3 的社交智能和 Theory of Mind 的能力，结果表明 GPT-3 在这些任务中存在严重的不足，进而考虑从人为中心的 NLP 方法入手可能会更加有效。