With the advancements in Large Language and Latent Diffusion models, brain decoding has achieved remarkable results in recent years. The works on the NSD dataset, with stimuli images from the COCO dataset, leverage the embeddings from the CLIP model for image reconstruction and GIT for captioning. However, the current captioning approach introduces the challenge of potential data contamination given that the GIT model was trained on the COCO dataset. In this work, we present an alternative method for decoding brain signals into image captions by predicting a DINOv2 model's embedding of an image from the corresponding fMRI signal and then providing its [CLS] token as the prefix to the GPT-2 language model which decreases computational requirements considerably. Additionally, instead of commonly used Linear Regression, we explore 3D Convolutional Neural Network mapping of fMRI signals to image embedding space for better accounting positional information of voxels.

本研究针对现有大脑解码方法在数据污染方面的局限性，提出了一种新颖的脑信号解码技术。通过预测DINOv2模型的图像嵌入来将fMRI信号解码为图像字幕，并以此作为GPT-2语言模型的前缀，显著减少了计算需求。同时，我们应用三维卷积神经网络映射fMRI信号，以更好地考虑体素的位置信息。此方法的提出有望提升大脑解码的精度和效率。

使用前缀语言模型将fMRI数据解码为字幕