Reconstructing human dynamic vision from brain activity is a challenging task
with great scientific significance. The difficulty stems from two primary
issues: (1) vision-processing mechanisms in the brain are highly intricate and
not fully revealed, making it challenging to directly learn a mapping between
fMRI and video; (2) the temporal resolution of fMRI is significantly lower than
that of natural videos. To overcome these issues, this paper propose a
two-stage model named Mind-Animator, which achieves state-of-the-art
performance on three public datasets. Specifically, during the fMRI-to-feature
stage, we decouple semantic, structural, and motion features from fMRI through
fMRI-vision-language tri-modal contrastive learning and sparse causal
attention. In the feature-to-video stage, these features are merged to videos
by an inflated Stable Diffusion. We substantiate that the reconstructed video
dynamics are indeed derived from fMRI, rather than hallucinations of the
generative model, through permutation tests. Additionally, the visualization of
voxel-wise and ROI-wise importance maps confirms the neurobiological
interpretability of our model.

提出了一种名为 Mind-Animator 的两阶段模型，通过脑活动数据可以重构人类动态视觉，并在三个公共数据集上达到最先进的性能，通过特征解耦和融合，证实了重构视频的神经学解释性。