Understanding the extent to which the perceptual world can be recovered from
language is a fundamental problem in cognitive science. We reformulate this
problem as that of distilling psychophysical information from text and show how
this can be done by combining large language models (LLMs) with a classic
psychophysical method based on similarity judgments. Specifically, we use the
prompt auto-completion functionality of GPT3, a state-of-the-art LLM, to elicit
similarity scores between stimuli and then apply multidimensional scaling to
uncover their underlying psychological space. We test our approach on six
perceptual domains and show that the elicited judgments strongly correlate with
human data and successfully recover well-known psychophysical structures such
as the color wheel and pitch spiral. We also explore meaningful divergences
between LLM and human representations. Our work showcases how combining
state-of-the-art machine models with well-known cognitive paradigms can shed
new light on fundamental questions in perception and language research.

使用大型语言模型结合经典的心理物理学方法，从文本中提炼出心理物理信息，然后应用多维尺度方法发现其潜在的心理空间。在六个知觉域上测试我们的方法，成功地拟合了文献中已知的心理物理结构。这项工作展示了如何将最先进的机器模型与经典的认知范式相结合，从而阐明感知和语言研究中的基本问题。

语言对感知的揭示：从大型语言模型中梳理心理物理知识

What Language Reveals about Perception: Distilling Psychophysical Knowledge from Large Language Models

Similarity judgments provide a well-established method for accessing mental
representations, with applications in psychology, neuroscience and machine
learning. However, collecting similarity judgments can be prohibitively
expensive for naturalistic datasets as the number of comparisons grows
quadratically in the number of stimuli. One way to tackle this problem is to
construct approximation procedures that rely on more accessible proxies for
predicting similarity. Here we leverage recent advances in language models and
online recruitment, proposing an efficient domain-general procedure for
predicting human similarity judgments based on text descriptions. Intuitively,
similar stimuli are likely to evoke similar descriptions, allowing us to use
description similarity to predict pairwise similarity judgments. Crucially, the
number of descriptions required grows only linearly with the number of stimuli,
drastically reducing the amount of data required. We test this procedure on six
datasets of naturalistic images and show that our models outperform previous
approaches based on visual information.

利用语言模型和在线招募，提出了一种基于文本描述的高效通用程序，以预测相似度判断，其数量只随刺激物数量线性增长，可极大减少数据需求，且在六个自然图像数据集上的结果优于基于视觉信息的先前方法。