Research efforts to understand neural signals have been ongoing for many
years, with visual decoding from fMRI signals attracting considerable
attention. Particularly, the advent of image diffusion models has advanced the
reconstruction of images from fMRI data significantly. However, existing
approaches often introduce inter- and intra- subject variations in the
reconstructed images, which can compromise accuracy. To address current
limitations in multi-subject brain decoding, we introduce a new Transformer
architecture called MindFormer. This model is specifically designed to generate
fMRI-conditioned feature vectors that can be used for conditioning Stable
Diffusion model. More specifically, MindFormer incorporates two key
innovations: 1) a novel training strategy based on the IP-Adapter to extract
semantically meaningful features from fMRI signals, and 2) a subject specific
token and linear layer that effectively capture individual differences in fMRI
signals while synergistically combines multi subject fMRI data for training.
Our experimental results demonstrate that Stable Diffusion, when integrated
with MindFormer, produces semantically consistent images across different
subjects. This capability significantly surpasses existing models in
multi-subject brain decoding. Such advancements not only improve the accuracy
of our reconstructions but also deepen our understanding of neural processing
variations among individuals.

通过与脑功能磁共振成像数据相关联的 MindFormer 模型，结合 Stable Diffusion 方法，能够在多个受试者上生成语义一致的图像，从而显著提高多个受试者的脑解码准确性。

MindFormer: 基于 fMRI 的多受试者脑解码的 Transformer 架构

MindFormer: A Transformer Architecture for Multi-Subject Brain Decoding  via fMRI

Decoding natural visual scenes from brain activity has flourished, with
extensive research in single-subject tasks and, however, less in cross-subject
tasks. Reconstructing high-quality images in cross-subject tasks is a
challenging problem due to profound individual differences between subjects and
the scarcity of data annotation. In this work, we proposed MindTuner for
cross-subject visual decoding, which achieves high-quality and rich-semantic
reconstructions using only 1 hour of fMRI training data benefiting from the
phenomena of visual fingerprint in the human visual system and a novel
fMRI-to-text alignment paradigm. Firstly, we pre-train a multi-subject model
among 7 subjects and fine-tune it with scarce data on new subjects, where LoRAs
with Skip-LoRAs are utilized to learn the visual fingerprint. Then, we take the
image modality as the intermediate pivot modality to achieve fMRI-to-text
alignment, which achieves impressive fMRI-to-text retrieval performance and
corrects fMRI-to-image reconstruction with fine-tuned semantics. The results of
both qualitative and quantitative analyses demonstrate that MindTuner surpasses
state-of-the-art cross-subject visual decoding models on the Natural Scenes
Dataset (NSD), whether using training data of 1 hour or 40 hours.

利用 1 小时的 fMRI 训练数据，结合人类视觉系统中的视觉指纹和一种新颖的 fMRI-to-text 对齐方法，我们提出了 MindTuner 用于跨主体视觉解码，实现了高质量和丰富语义的重建，结果表明 MindTuner 在自然场景数据集 (NSD) 上的表现优于最先进的跨主体视觉解码模型。

MindTuner：基于视觉指纹和语义纠正的跨个体视觉解码

MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and  Semantic Correction

Decoding visual stimuli from brain recordings aims to deepen our
understanding of the human visual system and build a solid foundation for
bridging human and computer vision through the Brain-Computer Interface.
However, reconstructing high-quality images with correct semantics from brain
recordings is a challenging problem due to the complex underlying
representations of brain signals and the scarcity of data annotations. In this
work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned
Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an
effective self-supervised representation of fMRI data using mask modeling in a
large latent space inspired by the sparse coding of information in the primary
visual cortex. Then by augmenting a latent diffusion model with
double-conditioning, we show that MinD-Vis can reconstruct highly plausible
images with semantically matching details from brain recordings using very few
paired annotations. We benchmarked our model qualitatively and quantitatively;
the experimental results indicate that our method outperformed state-of-the-art
in both semantic mapping (100-way semantic classification) and generation
quality (FID) by 66% and 41% respectively. An exhaustive ablation study was
also conducted to analyze our framework.

本文提出了一种使用双条件潜在扩散模型的 MinD-Vis：稀疏掩蔽脑建模方法，通过大型潜在空间中的掩蔽建模来学习 fMRI 数据的有效自我监督表示。实验证明，该方法在语义映射和生成质量方面都优于现有技术，并且只需很少的成对注释即可从脑记录中重建高度合理的图像。

超越大脑：使用稀疏掩码建模的条件扩散模型进行视觉解码

Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked  Modeling for Vision Decoding

What does human gaze reveal about a users' intents and to which extend can
these intents be inferred or even visualized? Gaze was proposed as an implicit
source of information to predict the target of visual search and, more
recently, to predict the object class and attributes of the search target. In
this work, we go one step further and investigate the feasibility of combining
recent advances in encoding human gaze information using deep convolutional
neural networks with the power of generative image models to visually decode,
i.e. create a visual representation of, the search target. Such visual decoding
is challenging for two reasons: 1) the search target only resides in the user's
mind as a subjective visual pattern, and can most often not even be described
verbally by the person, and 2) it is, as of yet, unclear if gaze fixations
contain sufficient information for this task at all. We show, for the first
time, that visual representations of search targets can indeed be decoded only
from human gaze fixations. We propose to first encode fixations into a semantic
representation and then decode this representation into an image. We evaluate
our method on a recent gaze dataset of 14 participants searching for clothing
in image collages and validate the model's predictions using two human studies.
Our results show that 62% (Chance level = 10%) of the time users were able to
select the categories of the decoded image right. In our second studies we show
the importance of a local gaze encoding for decoding visual search targets of
user

研究使用深度卷积神经网络和生成式图像模型将人类注视视线信息解码成视觉目标的图像表示，结果显示出 62% 的准确率