Reconstructing perceived natural images from fMRI signals is one of the most
engaging topics of neural decoding research. Prior studies had success in
reconstructing either the low-level image features or the semantic/high-level
aspects, but rarely both. In this study, we utilized an Instance-Conditioned
GAN (IC-GAN) model to reconstruct images from fMRI patterns with both accurate
semantic attributes and preserved low-level details. The IC-GAN model takes as
input a 119-dim noise vector and a 2048-dim instance feature vector extracted
from a target image via a self-supervised learning model (SwAV ResNet-50);
these instance features act as a conditioning for IC-GAN image generation,
while the noise vector introduces variability between samples. We trained ridge
regression models to predict instance features, noise vectors, and dense
vectors (the output of the first dense layer of the IC-GAN generator) of
stimuli from corresponding fMRI patterns. Then, we used the IC-GAN generator to
reconstruct novel test images based on these fMRI-predicted variables. The
generated images presented state-of-the-art results in terms of capturing the
semantic attributes of the original test images while remaining relatively
faithful to low-level image details. Finally, we use the learned regression
model and the IC-GAN generator to systematically explore and visualize the
semantic features that maximally drive each of several regions-of-interest in
the human brain.

本文研究使用 Instance-Conditioned GAN 模型从 fMRI 信号中重构自然图像，成功捕捉了语义信息和低层次图像细节，并使用回归模型探索了人类大脑感兴趣区域的语义特征。