In this paper, we introduce Recon3DMind, a groundbreaking task focused on
reconstructing 3D visuals from Functional Magnetic Resonance Imaging (fMRI)
signals. This represents a major step forward in cognitive neuroscience and
computer vision. To support this task, we present the fMRI-Shape dataset,
utilizing 360-degree view videos of 3D objects for comprehensive fMRI signal
capture. Containing 55 categories of common objects from daily life, this
dataset will bolster future research endeavors. We also propose MinD-3D, a
novel and effective three-stage framework that decodes and reconstructs the
brain's 3D visual information from fMRI signals. This method starts by
extracting and aggregating features from fMRI frames using a neuro-fusion
encoder, then employs a feature bridge diffusion model to generate
corresponding visual features, and ultimately recovers the 3D object through a
generative transformer decoder. Our experiments demonstrate that this method
effectively extracts features that are valid and highly correlated with visual
regions of interest (ROIs) in fMRI signals. Notably, it not only reconstructs
3D objects with high semantic relevance and spatial similarity but also
significantly deepens our understanding of the human brain's 3D visual
processing capabilities. Project page at: this https URL

从功能性磁共振成像（fMRI）信号中重建 3D 视觉，引入了 Recon3DMind 任务；使用 360 度全景视频的 fMRI-Shape 数据集支持这一任务；提出了 MinD-3D 三阶段框架，从 fMRI 信号中解码和重建大脑的 3D 视觉信息；通过有效提取与 fMRI 信号中的视觉兴趣区域高度相关的特征，实现了对高语义相关性和空间相似性的 3D 对象的重建，并深入了解人脑的 3D 视觉处理能力。