Humans possess the ability to identify and generalize relevant features of
natural objects, which aids them in various situations. To investigate this
phenomenon and determine the most effective representations for predicting
human behavior, we conducted two experiments involving category learning and
reward learning. Our experiments used realistic images as stimuli, and
participants were tasked with making accurate decisions based on novel stimuli
for all trials, thereby necessitating generalization. In both tasks, the
underlying rules were generated as simple linear functions using stimulus
dimensions extracted from human similarity judgments. Notably, participants
successfully identified the relevant stimulus features within a few trials,
demonstrating effective generalization. We performed an extensive model
comparison, evaluating the trial-by-trial predictive accuracy of diverse deep
learning models' representations of human choices. Intriguingly,
representations from models trained on both text and image data consistently
outperformed models trained solely on images, even surpassing models using the
features that generated the task itself. These findings suggest that
language-aligned visual representations possess sufficient richness to describe
human generalization in naturalistic settings and emphasize the role of
language in shaping human cognition.

本研究探讨类别学习和奖励学习实验中的泛化技能和预测人类行为的最有效表示方法，结果表明，深度学习模型从文本和图像数据中训练得到的表示方式优于仅从图像中训练得到的表示方式，强调了语言在塑造人类认知中的作用。