In the pursuit to understand the intricacies of human brain's visual
processing, reconstructing dynamic visual experiences from brain activities
emerges as a challenging yet fascinating endeavor. While recent advancements
have achieved success in reconstructing static images from non-invasive brain
recordings, the domain of translating continuous brain activities into video
format remains underexplored. In this work, we introduce NeuroCine, a novel
dual-phase framework to targeting the inherent challenges of decoding fMRI
data, such as noises, spatial redundancy and temporal lags. This framework
proposes spatial masking and temporal interpolation-based augmentation for
contrastive learning fMRI representations and a diffusion model enhanced by
dependent prior noise for video generation. Tested on a publicly available fMRI
dataset, our method shows promising results, outperforming the previous
state-of-the-art models by a notable margin of ${20.97\%}$, ${31.00\%}$ and
${12.30\%}$ respectively on decoding the brain activities of three subjects in
the fMRI dataset, as measured by SSIM. Additionally, our attention analysis
suggests that the model aligns with existing brain structures and functions,
indicating its biological plausibility and interpretability.

在理解人类大脑视觉处理的复杂性中，从大脑活动中重建动态视觉体验成为一项具有挑战性且引人入胜的努力。本研究引入了 NeuroCine，一种新颖的双相框架，针对解码 fMRI 数据中固有的挑战，如噪声、空间冗余和时间滞后。该框架通过对比学习 fMRI 表示进行空间遮蔽和时间插值增强，以及依赖性先验噪声增强的扩散模型用于视频生成。在一个公开可用的 fMRI 数据集上进行了测试，我们的方法显示出有希望的结果，通过 SSIM 测量，在解码三个主题的 fMRI 数据集中，与先前最先进的模型相比，分别提高了 20.97％，31.00％和 12.30％。此外，我们的注意力分析表明该模型与现有的大脑结构和功能相吻合，表明其具有生物学合理性和可解释性。