Reconstructing perceived natural images from fMRI signals is one of the most
engaging topics of neural decoding research. Prior studies had success in
reconstructing either the low-level image features or the semantic/high-level
aspects, but rarely both. In this study, we utilized an Instance-Conditioned
GAN (IC-GAN) model to reconstruct images from fMRI patterns with both accurate
semantic attributes and preserved low-level details. The IC-GAN model takes as
input a 119-dim noise vector and a 2048-dim instance feature vector extracted
from a target image via a self-supervised learning model (SwAV ResNet-50);
these instance features act as a conditioning for IC-GAN image generation,
while the noise vector introduces variability between samples. We trained ridge
regression models to predict instance features, noise vectors, and dense
vectors (the output of the first dense layer of the IC-GAN generator) of
stimuli from corresponding fMRI patterns. Then, we used the IC-GAN generator to
reconstruct novel test images based on these fMRI-predicted variables. The
generated images presented state-of-the-art results in terms of capturing the
semantic attributes of the original test images while remaining relatively
faithful to low-level image details. Finally, we use the learned regression
model and the IC-GAN generator to systematically explore and visualize the
semantic features that maximally drive each of several regions-of-interest in
the human brain.

本文研究使用 Instance-Conditioned GAN 模型从 fMRI 信号中重构自然图像，成功捕捉了语义信息和低层次图像细节，并使用回归模型探索了人类大脑感兴趣区域的语义特征。

使用实例条件 GAN 从 fMRI 图像和语义大脑探索中重建知觉图像

Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain  Exploration using Instance-Conditioned GANs

Medical image segmentation - the prerequisite of numerous clinical needs -
has been significantly prospered by recent advances in convolutional neural
networks (CNNs). However, it exhibits general limitations on modeling explicit
long-range relation, and existing cures, resorting to building deep encoders
along with aggressive downsampling operations, leads to redundant deepened
networks and loss of localized details. Hence, the segmentation task awaits a
better solution to improve the efficiency of modeling global contexts while
maintaining a strong grasp of low-level details. In this paper, we propose a
novel parallel-in-branch architecture, TransFuse, to address this challenge.
TransFuse combines Transformers and CNNs in a parallel style, where both global
dependency and low-level spatial details can be efficiently captured in a much
shallower manner. Besides, a novel fusion technique - BiFusion module is
created to efficiently fuse the multi-level features from both branches.
Extensive experiments demonstrate that TransFuse achieves the newest
state-of-the-art results on both 2D and 3D medical image sets including polyp,
skin lesion, hip, and prostate segmentation, with significant parameter
decrease and inference speed improvement.

本文介绍了用于医学图像分割的 TransFuse 并行网络结构，将 Transformers 和 CNNs 并置，以有效地捕捉全局依赖关系和低级别空间细节，且使用了一种新的特征融合方法 —BiFusion 模块来融合两个分支的多级特征。实验结果表明，TransFuse 在医学图像分割任务中的性能优于目前最先进的技术，同时在参数数量和推断速度上也有显著改进。