Analyzing and reconstructing visual stimuli from brain signals effectively
advances understanding of the human visual system. However, the EEG signals are
complex and contain a amount of noise. This leads to substantial limitations in
existing works of visual stimuli reconstruction from EEG, such as difficulties
in aligning EEG embeddings with the fine-grained semantic information and a
heavy reliance on additional large self-collected dataset for training. To
address these challenges, we propose a novel approach called BrainVis. Firstly,
we divide the EEG signals into various units and apply a self-supervised
approach on them to obtain EEG time-domain features, in an attempt to ease the
training difficulty. Additionally, we also propose to utilize the
frequency-domain features to enhance the EEG representations. Then, we
simultaneously align EEG time-frequency embeddings with the interpolation of
the coarse and fine-grained semantics in the CLIP space, to highlight the
primary visual components and reduce the cross-modal alignment difficulty.
Finally, we adopt the cascaded diffusion models to reconstruct images. Our
proposed BrainVis outperforms state of the arts in both semantic fidelity
reconstruction and generation quality. Notably, we reduce the training data
scale to 10% of the previous work.

通过将 EEG 信号分解成多个单元并自我监督学习获取时间域特征，利用频域特征增强 EEG 表示，将 EEG 时频嵌入与 CLIP 空间中的粗粒度语义插值对齐，使用级联扩散模型重建图像，我们提出的 BrainVis 在语义保真度重建和生成质量方面均优于现有技术，值得注意的是，我们将训练数据规模降低到以前的 10%。