Large language models (LLMs) are increasingly used in robotics, especially
for high-level action planning. Meanwhile, many robotics applications involve
human supervisors or collaborators. Hence, it is crucial for LLMs to generate
socially acceptable actions that align with people's preferences and values. In
this work, we test whether LLMs capture people's intuitions about behavior
judgments and communication preferences in human-robot interaction (HRI)
scenarios. For evaluation, we reproduce three HRI user studies, comparing the
output of LLMs with that of real participants. We find that GPT-4 strongly
outperforms other models, generating answers that correlate strongly with
users' answers in two studies $\unicode{x2014}$ the first study dealing with
selecting the most appropriate communicative act for a robot in various
situations ($r_s$ = 0.82), and the second with judging the desirability,
intentionality, and surprisingness of behavior ($r_s$ = 0.83). However, for the
last study, testing whether people judge the behavior of robots and humans
differently, no model achieves strong correlations. Moreover, we show that
vision models fail to capture the essence of video stimuli and that LLMs tend
to rate different communicative acts and behavior desirability higher than
people.

本研究探索了大型语言模型在人机交互领域的表现，对比实际参与者的答案，结果显示 GPT-4 在选择适当的沟通行为和评判行为可取性、意图和令人惊讶程度等方面表现较好，但在判断人机行为差异方面成绩不佳，同时指出视觉模型无法完全抓住视频刺激的本质，且大型语言模型对不同沟通行为的评分和行为可取性分数较高。