Prior work has offered evidence for functional localization in the brain;
different anatomical regions preferentially activate for certain types of
visual input. For example, the fusiform face area preferentially activates for
visual stimuli that include a face. However, the spectrum of visual semantics
is extensive, and only a few semantically-tuned patches of cortex have so far
been identified in the human brain. Using a multimodal (natural language and
image) neural network architecture (CLIP) we train a highly accurate
contrastive model that maps brain responses during naturalistic image viewing
to CLIP embeddings. We then use a novel adaptation of the DBSCAN clustering
algorithm to cluster the parameters of these participant-specific contrastive
models. This reveals what we call Shared Decodable Concepts (SDCs): clusters in
CLIP space that are decodable from common sets of voxels across multiple
participants.
Examining the images most and least associated with each SDC cluster gives us
additional insight into the semantic properties of each SDC. We note SDCs for
previously reported visual features (e.g. orientation tuning in early visual
cortex) as well as visual semantic concepts such as faces, places and bodies.
In cases where our method finds multiple clusters for a visuo-semantic concept,
the least associated images allow us to dissociate between confounding factors.
For example, we discovered two clusters of food images, one driven by color,
the other by shape. We also uncover previously unreported areas such as regions
of extrastriate body area (EBA) tuned for legs/hands and sensitivity to
numerosity in right intraparietal sulcus, and more. Thus, our
contrastive-learning methodology better characterizes new and existing
visuo-semantic representations in the brain by leveraging multimodal neural
network representations and a novel adaptation of clustering algorithms.

通过训练多模态（自然语言和图像）神经网络架构（CLIP），我们使用一种新颖的 DBSCAN 聚类算法的适应版来聚类这些参与者特定对比模型的参数，揭示了我们称之为共享解码概念（SDCs）的在多个参与者中通过共同一组体素解码的 CLIP 空间中的聚类。通过分析与每个 SDC 聚类最相关和最不相关的图像，我们对每个 SDC 的语义属性有了更深入的了解。因此，我们的对比学习方法通过利用多模态神经网络表示和一种新颖的聚类算法的适应版更好地描述了大脑中新的和现有的视觉 - 语义表示。

寻找脑中共同可解编码的概念及其否定的脸的对立是什么？

What's the Opposite of a Face? Finding Shared Decodable Concepts and  their Negations in the Brain

We introduce a method that takes advantage of high-quality pretrained
multimodal representations to explore fine-grained semantic networks in the
human brain. Previous studies have documented evidence of functional
localization in the brain, with different anatomical regions preferentially
activating for different types of sensory input. Many such localized structures
are known, including the fusiform face area and parahippocampal place area.
This raises the question of whether additional brain regions (or conjunctions
of brain regions) are also specialized for other important semantic concepts.
To identify such brain regions, we developed a data-driven approach to uncover
visual concepts that are decodable from a massive functional magnetic resonance
imaging (fMRI) dataset. Our analysis is broadly split into three sections.
First, a fully connected neural network is trained to map brain responses to
the outputs of an image-language foundation model, CLIP (Radford et al., 2021).
Subsequently, a contrastive-learning dimensionality reduction method reveals
the brain-decodable components of CLIP space. In the final section of our
analysis, we localize shared decodable concepts in the brain using a
voxel-masking optimization method to produce a shared decodable concept (SDC)
space. The accuracy of our procedure is validated by comparing it to previous
localization experiments that identify regions for faces, bodies, and places.
In addition to these concepts, whose corresponding brain regions were already
known, we localize novel concept representations which are shared across
participants to other areas of the human brain. We also demonstrate how this
method can be used to inspect fine-grained semantic networks for individual
participants. We envisage that this extensible method can also be adapted to
explore other questions at the intersection of AI and neuroscience.

本研究介绍了一种基于高质量预训练多模态表示利用功能磁共振成像技术探索人类大脑微观语义网络的方法，并描述了该方法在检测面孔、身体和空间等重要语义概念的表现及其在个体参与者中的应用。