Decoding natural visual scenes from brain activity has flourished, with
extensive research in single-subject tasks and, however, less in cross-subject
tasks. Reconstructing high-quality images in cross-subject tasks is a
challenging problem due to profound individual differences between subjects and
the scarcity of data annotation. In this work, we proposed MindTuner for
cross-subject visual decoding, which achieves high-quality and rich-semantic
reconstructions using only 1 hour of fMRI training data benefiting from the
phenomena of visual fingerprint in the human visual system and a novel
fMRI-to-text alignment paradigm. Firstly, we pre-train a multi-subject model
among 7 subjects and fine-tune it with scarce data on new subjects, where LoRAs
with Skip-LoRAs are utilized to learn the visual fingerprint. Then, we take the
image modality as the intermediate pivot modality to achieve fMRI-to-text
alignment, which achieves impressive fMRI-to-text retrieval performance and
corrects fMRI-to-image reconstruction with fine-tuned semantics. The results of
both qualitative and quantitative analyses demonstrate that MindTuner surpasses
state-of-the-art cross-subject visual decoding models on the Natural Scenes
Dataset (NSD), whether using training data of 1 hour or 40 hours.

利用 1 小时的 fMRI 训练数据，结合人类视觉系统中的视觉指纹和一种新颖的 fMRI-to-text 对齐方法，我们提出了 MindTuner 用于跨主体视觉解码，实现了高质量和丰富语义的重建，结果表明 MindTuner 在自然场景数据集 (NSD) 上的表现优于最先进的跨主体视觉解码模型。