Embodied language comprehension emphasizes that language understanding is not
solely a matter of mental processing in the brain but also involves
interactions with the physical and social environment. With the explosive
growth of Large Language Models (LLMs) and their already ubiquitous presence in
our daily lives, it is becoming increasingly necessary to verify their
real-world understanding. Inspired by cognitive theories, we propose POSQA: a
Physical Object Size Question Answering dataset with simple size comparison
questions to examine the extremity and analyze the potential mechanisms of the
embodied comprehension of the latest LLMs.
We show that even the largest LLMs today perform poorly under the zero-shot
setting. We then push their limits with advanced prompting techniques and
external knowledge augmentation. Furthermore, we investigate whether their
real-world comprehension primarily derives from contextual information or
internal weights and analyse the impact of prompt formats and report bias of
different objects. Our results show that real-world understanding that LLMs
shaped from textual data can be vulnerable to deception and confusion by the
surface form of prompts, which makes it less aligned with human behaviours.

通过提出 POSQA 数据集，我们研究了最新的大型语言模型的体现性理解，并发现即使是目前最大的语言模型在零样本设置下表现不佳，进一步通过高级提示技术和外部知识增强来提升它们的性能。我们调查了他们对真实世界理解的主要来源以及提示格式的影响，结果表明基于文本数据形成的语言模型对提示的表面形式容易受到欺骗和困惑，导致其与人类行为的契合性降低。