Reconstructing visual stimuli from brain recordings has been a meaningful and
challenging task. Especially, the achievement of precise and controllable image
reconstruction bears great significance in propelling the progress and
utilization of brain-computer interfaces. Despite the advancements in complex
image reconstruction techniques, the challenge persists in achieving a cohesive
alignment of both semantic (concepts and objects) and structure (position,
orientation, and size) with the image stimuli. To address the aforementioned
issue, we propose a two-stage image reconstruction model called MindDiffuser.
In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings
decoded from fMRI are put into Stable Diffusion, which yields a preliminary
image that contains semantic information. In Stage 2, we utilize the CLIP
visual feature decoded from fMRI as supervisory information, and continually
adjust the two feature vectors decoded in Stage 1 through backpropagation to
align the structural information. The results of both qualitative and
quantitative analyses demonstrate that our model has surpassed the current
state-of-the-art models on Natural Scenes Dataset (NSD). The subsequent
experimental findings corroborate the neurobiological plausibility of the
model, as evidenced by the interpretability of the multimodal feature employed,
which align with the corresponding brain responses.

我们提出了一种名为 MindDiffuser 的两阶段图像重建模型，通过使用 fMRI 进行前向估计和反向传播来实现想定图像的语义和结构信息的精确对齐，实验结果表明该模型在自然场景数据集上超过了现有的最先进模型，并且与相应的大脑反应呈现的多模态特征解释力一致，从而证实其神经生物学的合理性。