Prior work has offered evidence for functional localization in the brain;
different anatomical regions preferentially activate for certain types of
visual input. For example, the fusiform face area preferentially activates for
visual stimuli that include a face. However, the spectrum of visual semantics
is extensive, and only a few semantically-tuned patches of cortex have so far
been identified in the human brain. Using a multimodal (natural language and
image) neural network architecture (CLIP) we train a highly accurate
contrastive model that maps brain responses during naturalistic image viewing
to CLIP embeddings. We then use a novel adaptation of the DBSCAN clustering
algorithm to cluster the parameters of these participant-specific contrastive
models. This reveals what we call Shared Decodable Concepts (SDCs): clusters in
CLIP space that are decodable from common sets of voxels across multiple
participants.
Examining the images most and least associated with each SDC cluster gives us
additional insight into the semantic properties of each SDC. We note SDCs for
previously reported visual features (e.g. orientation tuning in early visual
cortex) as well as visual semantic concepts such as faces, places and bodies.
In cases where our method finds multiple clusters for a visuo-semantic concept,
the least associated images allow us to dissociate between confounding factors.
For example, we discovered two clusters of food images, one driven by color,
the other by shape. We also uncover previously unreported areas such as regions
of extrastriate body area (EBA) tuned for legs/hands and sensitivity to
numerosity in right intraparietal sulcus, and more. Thus, our
contrastive-learning methodology better characterizes new and existing
visuo-semantic representations in the brain by leveraging multimodal neural
network representations and a novel adaptation of clustering algorithms.

通过训练多模态（自然语言和图像）神经网络架构（CLIP），我们使用一种新颖的 DBSCAN 聚类算法的适应版来聚类这些参与者特定对比模型的参数，揭示了我们称之为共享解码概念（SDCs）的在多个参与者中通过共同一组体素解码的 CLIP 空间中的聚类。通过分析与每个 SDC 聚类最相关和最不相关的图像，我们对每个 SDC 的语义属性有了更深入的了解。因此，我们的对比学习方法通过利用多模态神经网络表示和一种新颖的聚类算法的适应版更好地描述了大脑中新的和现有的视觉 - 语义表示。