Research in decoding visual information from the brain, particularly through
the non-invasive fMRI method, is rapidly progressing. The challenge arises from
the limited data availability and the low signal-to-noise ratio of fMRI
signals, leading to a low-precision task of fMRI-to-image retrieval.
State-of-the-art MindEye remarkably improves fMRI-to-image retrieval
performance by leveraging a deep MLP with a high parameter count orders of
magnitude, i.e., a 996M MLP Backbone per subject, to align fMRI embeddings to
the final hidden layer of CLIP's vision transformer. However, significant
individual variations exist among subjects, even within identical experimental
setups, mandating the training of subject-specific models. The substantial
parameters pose significant challenges in deploying fMRI decoding on practical
devices, especially with the necessitating of specific models for each subject.
To this end, we propose Lite-Mind, a lightweight, efficient, and versatile
brain representation network based on discrete Fourier transform, that
efficiently aligns fMRI voxels to fine-grained information of CLIP. Our
experiments demonstrate that Lite-Mind achieves an impressive 94.3%
fMRI-to-image retrieval accuracy on the NSD dataset for Subject 1, with 98.7%
fewer parameters than MindEye. Lite-Mind is also proven to be able to be
migrated to smaller brain datasets and establishes a new state-of-the-art for
zero-shot classification on the GOD dataset. The code is available at
this https URL

通过 Lite-Mind，基于离散傅里叶变换的轻量级、高效且通用的脑表示网络，我们成功地在 NSD 数据集的 Subject 1 上实现了 94.3% 的 fMRI 到图像检索准确率，比 MindEye 少 98.7% 的参数，并且在 GOD 数据集上实现了零样本分类的新的最先进成果。