We argue that there are certain properties of language that our current large
language models (LLMs) don't learn. We present an empirical investigation of
visual-auditory properties of language through a series of tasks, termed
H-TEST. This benchmark highlights a fundamental gap between human linguistic
comprehension, which naturally integrates sensory experiences, and the
sensory-deprived processing capabilities of LLMs. In support of our hypothesis,
1. deliberate reasoning (Chain-of-Thought), 2. few-shot examples, or 3.
stronger LLM from the same model family (LLaMA 2 13B -> LLaMA 2 70B) do not
trivially bring improvements in H-TEST performance. Therefore, we make a
particular connection to the philosophical case of Mary, who learns about the
world in a sensory-deprived environment (Jackson, 1986). Our experiments show
that some of the strongest proprietary LLMs stay near random chance baseline
accuracy of 50%, highlighting the limitations of knowledge acquired in the
absence of sensory experience.

我们通过一系列任务（H-TEST）对语言的视听属性进行了实证调查，发现当前大规模语言模型（LLMs）存在无法学习的语言属性。我们的实验证明，常识推理、少样本示例或同一模型系列的更强大的语言模型不能显著改善 H-TEST 性能，这与人类的语言理解存在本质差距，并突出显示了在缺乏感知经验的情况下获取知识的局限性。

语言模型未学习的任务

Tasks That Language Models Don't Learn

Prominent questions about the role of sensory vs. linguistic input in the way
we acquire and use language have been extensively studied in the
psycholinguistic literature. However, the relative effect of various factors in
a person's overall experience on their linguistic system remains unclear. We
study this question by making a step forward towards a better understanding of
the conceptual perception of colors by color-blind individuals, as reflected in
their spontaneous linguistic productions. Using a novel and carefully curated
dataset, we show that red-green color-blind speakers use the "red" and "green"
color terms in less predictable contexts, and in linguistic environments
evoking mental image to a lower extent, when compared to their normal-sighted
counterparts. These findings shed some new and interesting light on the role of
sensory experience on our linguistic system.

研究了色盲患者对红绿色观念知觉的影响，发现他们的语言产出中红色和绿色词汇使用的可预测性较低且不常出现在心理图像引发的语言环境中，揭示了感官经验对于语言系统的影响。