Deciphering visual content from functional Magnetic Resonance Imaging (fMRI)
helps illuminate the human vision system. However, the scarcity of fMRI data
and noise hamper brain decoding model performance. Previous approaches
primarily employ subject-specific models, sensitive to training sample size. In
this paper, we explore a straightforward but overlooked solution to address
data scarcity. We propose shallow subject-specific adapters to map
cross-subject fMRI data into unified representations. Subsequently, a shared
deeper decoding model decodes cross-subject features into the target feature
space. During training, we leverage both visual and textual supervision for
multi-modal brain decoding. Our model integrates a high-level perception
decoding pipeline and a pixel-wise reconstruction pipeline guided by high-level
perceptions, simulating bottom-up and top-down processes in neuroscience.
Empirical experiments demonstrate robust neural representation learning across
subjects for both pipelines. Moreover, merging high-level and low-level
information improves both low-level and high-level reconstruction metrics.
Additionally, we successfully transfer learned general knowledge to new
subjects by training new adapters with limited training data. Compared to
previous state-of-the-art methods, notably pre-training-based methods (Mind-Vis
and fMRI-PTE), our approach achieves comparable or superior results across
diverse tasks, showing promise as an alternative method for cross-subject fMRI
data pre-training. Our code and pre-trained weights will be publicly released
at this https URL

我们提出了一种新的方法来解决功能磁共振成像（fMRI）数据稀缺和噪声干扰脑解码模型性能的问题，通过浅层主体特定适配器将跨学科的 fMRI 数据映射到统一的表示中，然后使用共享的深层解码模型将跨学科特征解码为目标特征空间，利用视觉和文本监督进行多模态脑解码的训练，实验证明了我们模型在所有科目中的鲁棒神经表示学习，并且将高级和低级信息相结合可以改善重建度量。