We argue that there are certain properties of language that our current large
language models (LLMs) don't learn. We present an empirical investigation of
visual-auditory properties of language through a series of tasks, termed
H-TEST. This benchmark highlights a fundamental gap between human linguistic
comprehension, which naturally integrates sensory experiences, and the
sensory-deprived processing capabilities of LLMs. In support of our hypothesis,
1. deliberate reasoning (Chain-of-Thought), 2. few-shot examples, or 3.
stronger LLM from the same model family (LLaMA 2 13B -> LLaMA 2 70B) do not
trivially bring improvements in H-TEST performance. Therefore, we make a
particular connection to the philosophical case of Mary, who learns about the
world in a sensory-deprived environment (Jackson, 1986). Our experiments show
that some of the strongest proprietary LLMs stay near random chance baseline
accuracy of 50%, highlighting the limitations of knowledge acquired in the
absence of sensory experience.

我们通过一系列任务（H-TEST）对语言的视听属性进行了实证调查，发现当前大规模语言模型（LLMs）存在无法学习的语言属性。我们的实验证明，常识推理、少样本示例或同一模型系列的更强大的语言模型不能显著改善 H-TEST 性能，这与人类的语言理解存在本质差距，并突出显示了在缺乏感知经验的情况下获取知识的局限性。