Decoding natural visual scenes from brain activity has flourished, with
extensive research in single-subject tasks and, however, less in cross-subject
tasks. Reconstructing high-quality images in cross-subject tasks is a
challenging problem due to profound individual differences between subjects and
the scarcity of data annotation. In this work, we proposed MindTuner for
cross-subject visual decoding, which achieves high-quality and rich-semantic
reconstructions using only 1 hour of fMRI training data benefiting from the
phenomena of visual fingerprint in the human visual system and a novel
fMRI-to-text alignment paradigm. Firstly, we pre-train a multi-subject model
among 7 subjects and fine-tune it with scarce data on new subjects, where LoRAs
with Skip-LoRAs are utilized to learn the visual fingerprint. Then, we take the
image modality as the intermediate pivot modality to achieve fMRI-to-text
alignment, which achieves impressive fMRI-to-text retrieval performance and
corrects fMRI-to-image reconstruction with fine-tuned semantics. The results of
both qualitative and quantitative analyses demonstrate that MindTuner surpasses
state-of-the-art cross-subject visual decoding models on the Natural Scenes
Dataset (NSD), whether using training data of 1 hour or 40 hours.

利用 1 小时的 fMRI 训练数据，结合人类视觉系统中的视觉指纹和一种新颖的 fMRI-to-text 对齐方法，我们提出了 MindTuner 用于跨主体视觉解码，实现了高质量和丰富语义的重建，结果表明 MindTuner 在自然场景数据集 (NSD) 上的表现优于最先进的跨主体视觉解码模型。

MindTuner：基于视觉指纹和语义纠正的跨个体视觉解码

MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and  Semantic Correction

Reconstructions of visual perception from brain activity have improved
tremendously, but the practical utility of such methods has been limited. This
is because such models are trained independently per subject where each subject
requires dozens of hours of expensive fMRI training data to attain high-quality
results. The present work showcases high-quality reconstructions using only 1
hour of fMRI training data. We pretrain our model across 7 subjects and then
fine-tune on minimal data from a new subject. Our novel functional alignment
procedure linearly maps all brain data to a shared-subject latent space,
followed by a shared non-linear mapping to CLIP image space. We then map from
CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP
latents as inputs instead of text. This approach improves out-of-subject
generalization with limited training data and also attains state-of-the-art
image retrieval and reconstruction metrics compared to single-subject
approaches. MindEye2 demonstrates how accurate reconstructions of perception
are possible from a single visit to the MRI facility. All code is available on
GitHub.

使用仅 1 小时的 fMRI 训练数据展示了高质量的知觉重建，我们通过在 7 个受试者上进行预训练并在新受试者上进行微调的方法，通过功能对齐和 CLIP 图像空间的映射，实现了对知觉的准确重建。