Decoding of seen visual contents with non-invasive brain recordings has
important scientific and practical values. Efforts have been made to recover
the seen images from brain signals. However, most existing approaches cannot
faithfully reflect the visual contents due to insufficient image quality or
semantic mismatches. Compared with reconstructing pixel-level visual images,
speaking is a more efficient and effective way to explain visual information.
Here we introduce a non-invasive neural decoder, termed as MindGPT, which
interprets perceived visual stimuli into natural languages from fMRI signals.
Specifically, our model builds upon a visually guided neural encoder with a
cross-attention mechanism, which permits us to guide latent neural
representations towards a desired language semantic direction in an end-to-end
manner by the collaborative use of the large language model GPT. By doing so,
we found that the neural representations of the MindGPT are explainable, which
can be used to evaluate the contributions of visual properties to language
semantics. Our experiments show that the generated word sequences truthfully
represented the visual information (with essential details) conveyed in the
seen stimuli. The results also suggested that with respect to language decoding
tasks, the higher visual cortex (HVC) is more semantically informative than the
lower visual cortex (LVC), and using only the HVC can recover most of the
semantic information. The code of the MindGPT model will be publicly available
at this https URL

通过 fMRI 信号解释感知到的视觉刺激为自然语言的一种非侵入性神经解码器 MindGPT，该模型建立在具有交叉注意机制的视觉引导神经编码器上，并通过大型语言模型 GPT 的协同使用来将潜在的神经表示引导到期望的语义方向，从而使 MindGPT 的神经表示具有可解释性，可以用来评估视觉属性对语义的贡献。实验证明生成的词序列真实地代表了感知刺激中传达的视觉信息（具有关键细节），结果还表明，对于语言解码任务，高级视觉皮层（HVC）比低级视觉皮层（LVC）更具语义信息，仅使用 HVC 就可以恢复大部分语义信息。MindGPT 模型的代码将公开发布在此 https URL 上。