The reconstruction of images observed by subjects from fMRI data collected
during visual stimuli has made significant strides in the past decade, thanks
to the availability of extensive fMRI datasets and advancements in generative
models for image generation. However, the application of visual reconstruction
has remained limited. Reconstructing visual imagination presents a greater
challenge, with potentially revolutionary applications ranging from aiding
individuals with disabilities to verifying witness accounts in court. The
primary hurdles in this field are the absence of data collection protocols for
visual imagery and the lack of datasets on the subject. Traditionally,
fMRI-to-image relies on data collected from subjects exposed to visual stimuli,
which poses issues for generating visual imagery based on the difference of
brain activity between visual stimulation and visual imagery. For the first
time, we have compiled a substantial dataset (around 6h of scans) on visual
imagery along with a proposed data collection protocol. We then train a
modified version of an fMRI-to-image model and demonstrate the feasibility of
reconstructing images from two modes of imagination: from memory and from pure
imagination. This marks an important step towards creating a technology that
allow direct reconstruction of visual imagery.

通过 fMRI 数据和修改的 fMRI-to-image 模型，我们首次在记忆和纯想象两种模式下，从视觉想象中重建图像的可行性，从而迈出了直接重建视觉想象的技术的重要一步。

脑力到图像：将大脑的视觉心理想象从 fMRI 投影出来

Mind-to-Image: Projecting Visual Mental Imagination of the Brain from  fMRI

In this paper, we propose \textbf{UniCode}, a novel approach within the
domain of multimodal large language models (MLLMs) that learns a unified
codebook to efficiently tokenize visual, text, and potentially other types of
signals. This innovation addresses a critical limitation in existing MLLMs:
their reliance on a text-only codebook, which restricts MLLM's ability to
generate images and texts in a multimodal context. Towards this end, we propose
a language-driven iterative training paradigm, coupled with an in-context
pre-training task we term ``image decompression'', enabling our model to
interpret compressed visual data and generate high-quality images.The unified
codebook empowers our model to extend visual instruction tuning to
non-linguistic generation tasks. Moreover, UniCode is adaptable to diverse
stacked quantization approaches in order to compress visual signals into a more
compact token representation. Despite using significantly fewer parameters and
less data during training, Unicode demonstrates promising capabilities in
visual reconstruction and generation. It also achieves performances comparable
to leading MLLMs across a spectrum of VQA benchmarks.

提出了一种名为 UniCode 的新方法，通过学习统一的码书，能够在多模态的大型语言模型中有效地对视觉、文本和其他类型的信号进行标记化，通过使用语言驱动的迭代训练方法和图像解压缩预训练任务，我们的模型能够解释压缩的视觉数据并生成高质量图像。UniCode 可适应各种堆叠的量化方法，将视觉信号压缩为更紧凑的标记表示。尽管在训练过程中使用的参数和数据少得多，但 UniCode 在视觉重建和生成方面展示出有希望的能力，并在各种 VQA 基准测试中实现了与领先的 MLLM 相当的性能。