High-level visual brain regions contain subareas in which neurons appear to
respond more strongly to examples of a particular semantic category, like faces
or bodies, rather than objects. However, recent work has shown that while this
finding holds on average, some out-of-category stimuli also activate neurons in
these regions. This may be due to visual features common among the preferred
class also being present in other images. Here, we propose a
deep-learning-based approach for visualizing these features. For each neuron,
we identify relevant visual features driving its selectivity by modelling
responses to images based on latent activations of a deep neural network. Given
an out-of-category image which strongly activates the neuron, our method first
identifies a reference image from the preferred category yielding a similar
feature activation pattern. We then backpropagate latent activations of both
images to the pixel level, while enhancing the identified shared dimensions and
attenuating non-shared features. The procedure highlights image regions
containing shared features driving responses of the model neuron. We apply the
algorithm to novel recordings from body-selective regions in macaque IT cortex
in order to understand why some images of objects excite these neurons.
Visualizations reveal object parts which resemble parts of a macaque body,
shedding light on neural preference of these objects.

使用基于深度学习的方法，通过对图像的激活进行建模，可以揭示大脑视觉区域中神经元的选择性和共享特征，进而理解为什么一些物体图像会激发这些神经元的反应。

共享特征可视化的并行反向传播

Parallel Backpropagation for Shared-Feature Visualization

Previous studies have shown that it is possible to map brain activation data
of subjects viewing images onto the feature representation space of not only
vision models (modality-specific decoding) but also language models
(cross-modal decoding). In this work, we introduce and use a new large-scale
fMRI dataset (~8,500 trials per subject) of people watching both images and
text descriptions of such images. This novel dataset enables the development of
modality-agnostic decoders: a single decoder that can predict which stimulus a
subject is seeing, irrespective of the modality (image or text) in which the
stimulus is presented. We train and evaluate such decoders to map brain signals
onto stimulus representations from a large range of publicly available vision,
language and multimodal (vision+language) models. Our findings reveal that (1)
modality-agnostic decoders perform as well as (and sometimes even better than)
modality-specific decoders (2) modality-agnostic decoders mapping brain data
onto representations from unimodal models perform as well as decoders relying
on multimodal representations (3) while language and low-level visual
(occipital) brain regions are best at decoding text and image stimuli,
respectively, high-level visual (temporal) regions perform well on both
stimulus types.

通过大规模的功能性磁共振成像 (fMRI) 数据集，本研究提出并使用了一种新的模态不可知解码器，能够将大范围的视觉、语言和多模态模型的脑信号映射到刺激表示上，发现模态不可知解码器表现与模态特定解码器相当甚至更好，并且单模态解码器表现与多模态解码器相当，同时高级的视觉区域可以很好地解码文本和图像刺激。