Many recent studies have shown that the perception of speech can be decoded
from brain signals and subsequently reconstructed as continuous language.
However, there is a lack of neurological basis for how the semantic information
embedded within brain signals can be used more effectively to guide language
reconstruction. The theory of predictive coding suggests that human brain
naturally engages in continuously predicting future word representations that
span multiple timescales. This implies that the decoding of brain signals could
potentially be associated with a predictable future. To explore the predictive
coding theory within the context of language reconstruction, this paper
proposes a novel model \textsc{PredFT} for jointly modeling neural decoding and
brain prediction. It consists of a main decoding network for language
reconstruction and a side network for predictive coding. The side network
obtains brain predictive coding representation from related brain regions of
interest with a multi-head self-attention module. This representation is fused
into the main decoding network with cross-attention to facilitate the language
models' generation process. Experiments are conducted on the largest
naturalistic language comprehension fMRI dataset Narratives. \textsc{PredFT}
achieves current state-of-the-art decoding performance with a maximum BLEU-1
score of $27.8\%$.

提出了一种用于神经解码和脑预测的新模型 	extsc {PredFT}，它结合了主要解码网络和用于预测编码的辅助网络，并通过交叉注意力将脑预测编码表示融入主要解码网络以促进语言模型的生成过程。在最大 BLEU-1 分数为 27.8% 的自然语言理解 fMRI 数据集 Narratives 上进行了实验，取得了当前最先进的解码性能。

利用 fMRI 数据的大脑预测编码进行语言重建

Language Reconstruction with Brain Predictive Coding from fMRI Data

Referring image segmentation aims to segment an object referred to by natural
language expression from an image. The primary challenge lies in the efficient
propagation of fine-grained semantic information from textual features to
visual features. Many recent works utilize a Transformer to address this
challenge. However, conventional transformer decoders can distort linguistic
information with deeper layers, leading to suboptimal results. In this paper,
we introduce CRFormer, a model that iteratively calibrates multi-modal features
in the transformer decoder. We start by generating language queries using
vision features, emphasizing different aspects of the input language. Then, we
propose a novel Calibration Decoder (CDec) wherein the multi-modal features can
iteratively calibrated by the input language features. In the Calibration
Decoder, we use the output of each decoder layer and the original language
features to generate new queries for continuous calibration, which gradually
updates the language features. Based on CDec, we introduce a Language
Reconstruction Module and a reconstruction loss. This module leverages queries
from the final layer of the decoder to reconstruct the input language and
compute the reconstruction loss. This can further prevent the language
information from being lost or distorted. Our experiments consistently show the
superior performance of our approach across RefCOCO, RefCOCO+, and G-Ref
datasets compared to state-of-the-art methods.

利用 CRFormer 模型，通过连续校准多模态特征，引入语言重建模块和重建损失，解决自然语言表达的图像分割问题，并在多个数据集上表现优于现有方法。