Recent fMRI-to-image approaches mainly focused on associating fMRI signals
with specific conditions of pre-trained diffusion models. These approaches,
while producing high-quality images, capture only a limited aspect of the
complex information in fMRI signals and offer little detailed control over
image creation. In contrast, this paper proposes to directly modulate the
generation process of diffusion models using fMRI signals. Our approach,
NeuroPictor, divides the fMRI-to-image process into three steps: i) fMRI
calibrated-encoding, to tackle multi-individual pre-training for a shared
latent space to minimize individual difference and enable the subsequent
cross-subject training; ii) fMRI-to-image cross-subject pre-training,
perceptually learning to guide diffusion model with high- and low-level
conditions across different individuals; iii) fMRI-to-image single-subject
refining, similar with step ii but focus on adapting to particular individual.
NeuroPictor extracts high-level semantic features from fMRI signals that
characterizing the visual stimulus and incrementally fine-tunes the diffusion
model with a low-level manipulation network to provide precise structural
instructions. By training with over 60,000 fMRI-image pairs from various
individuals, our model enjoys superior fMRI-to-image decoding capacity,
particularly in the within-subject setting, as evidenced in benchmark datasets.
Project page: this https URL

本研究提出了一种名为 NeuroPictor 的方法，通过直接调节扩散模型的生成过程，利用 fMRI 信号从中提取高级语义特征，并使用低级操作网络对扩散模型进行精细调整，从而实现准确的结构指令，进一步提高 fMRI 到图像的解码能力，尤其在同一受试者的情境中。