Large language models (LLMs) like ChatGPT have shown significant advancements
across diverse natural language understanding (NLU) tasks, including
intelligent dialogue and autonomous agents. Yet, lacking widely acknowledged
testing mechanisms, answering `whether LLMs are stochastic parrots or genuinely
comprehend the world' remains unclear, fostering numerous studies and sparking
heated debates. Prevailing research mainly focuses on surface-level NLU,
neglecting fine-grained explorations. However, such explorations are crucial
for understanding their unique comprehension mechanisms, aligning with human
cognition, and finally enhancing LLMs' general NLU capacities. To address this
gap, our study delves into LLMs' nuanced semantic comprehension capabilities,
particularly regarding common words with uncommon meanings. The idea stems from
foundational principles of human communication within psychology, which
underscore accurate shared understandings of word semantics. Specifically, this
paper presents the innovative construction of a Lexical Semantic Comprehension
(LeSC) dataset with novel evaluation metrics, the first benchmark encompassing
both fine-grained and cross-lingual dimensions. Introducing models of both
open-source and closed-source, varied scales and architectures, our extensive
empirical experiments demonstrate the inferior performance of existing models
in this basic lexical-meaning understanding task. Notably, even the
state-of-the-art LLMs GPT-4 and GPT-3.5 lag behind 16-year-old humans by 3.9%
and 22.3%, respectively. Additionally, multiple advanced prompting techniques
and retrieval-augmented generation are also introduced to help alleviate this
trouble, yet limitations persist. By highlighting the above critical
shortcomings, this research motivates further investigation and offers novel
insights for developing more intelligent LLMs.

通过创新构建一个包含细粒度和跨语言维度的词汇语义理解数据集，本研究揭示了大语言模型在基本词汇意义理解任务上的性能不佳，甚至落后于 16 岁的人类 3.9% 和 22.3% 分别。这突显了其关键不足，并激发了进一步研究和开发更智能的大语言模型的新见解。