Deciphering natural language from brain activity through non-invasive devices
remains a formidable challenge. Previous non-invasive decoders either require
multiple experiments with identical stimuli to pinpoint cortical regions and
enhance signal-to-noise ratios in brain activity, or they are limited to
discerning basic linguistic elements such as letters and words. We propose a
novel approach to decoding continuous language from single-trial non-invasive
fMRI recordings, in which a three-dimensional convolutional network augmented
with information bottleneck is developed to automatically identify responsive
voxels to stimuli, and a character-based decoder is designed for the semantic
reconstruction of continuous language characterized by inherent character
structures. The resulting decoder can produce intelligible textual sequences
that faithfully capture the meaning of perceived speech both within and across
subjects, while existing decoders exhibit significantly inferior performance in
cross-subject contexts. The ability to decode continuous language from single
trials across subjects demonstrates the promising applications of non-invasive
language brain-computer interfaces in both healthcare and neuroscience.

通过非侵入式设备从大脑活动中解读自然语言仍然是一个巨大的挑战，本文提出了一种新颖的方法，通过三维卷积网络和信息瓶颈技术对单次非侵入性功能磁共振成像记录进行连续语言解码，该解码器能够生成捕捉到感知语音含义的可理解文本序列，并在跨受试者场景中表现出明显优于现有解码器的性能，展示了非侵入式语言脑机接口在医疗保健和神经科学领域的应用前景。

从非侵入式脑电记录中解码连续字符语言

Decoding Continuous Character-based Language from Non-invasive Brain  Recordings

Reconstructing natural language from non-invasive electroencephalography
(EEG) holds great promise as a language decoding technology for brain-computer
interfaces (BCIs). However, EEG-based language decoding is still in its nascent
stages, facing several technical issues such as: 1) Absence of a hybrid
strategy that can effectively integrate cross-modality (between EEG and text)
self-learning with intra-modality self-reconstruction of EEG features or
textual sequences; 2) Under-utilization of large language models (LLMs) to
enhance EEG-based language decoding. To address above issues, we propose the
Contrastive EEG-Text Masked Autoencoder (CET-MAE), a novel model that
orchestrates compound self-supervised learning across and within EEG and text
through a dedicated multi-stream encoder. Furthermore, we develop a framework
called E2T-PTR (EEG-to-Text decoding using Pretrained Transferable
Representations), which leverages pre-trained modules alongside the EEG stream
from CET-MAE and further enables an LLM (specifically BART) to decode text from
EEG sequences. Comprehensive experiments conducted on the popular text-evoked
EEG database, ZuCo, demonstrate the superiority of E2T-PTR, which outperforms
the state-of-the-art in ROUGE-1 F1 and BLEU-4 scores by 8.34% and 32.21%,
respectively. These results indicate significant advancements in the field and
underscores the proposed framework's potential to enable more powerful and
widespread BCI applications.

提出了一种新颖的 Contrastive EEG-Text Masked Autoencoder 模型，并借助预训练模块以及 EEG 流实现了 EEG 到文本的解码，实验结果表明在文本激发的 EEG 数据库上，该模型在 ROUGE-1 F1 和 BLEU-4 得分上分别超过现有技术 8.34％和 32.21％，这显示了该框架在大脑计算界面应用领域的潜力。