Deciphering natural language from brain activity through non-invasive devices
remains a formidable challenge. Previous non-invasive decoders either require
multiple experiments with identical stimuli to pinpoint cortical regions and
enhance signal-to-noise ratios in brain activity, or they are limited to
discerning basic linguistic elements such as letters and words. We propose a
novel approach to decoding continuous language from single-trial non-invasive
fMRI recordings, in which a three-dimensional convolutional network augmented
with information bottleneck is developed to automatically identify responsive
voxels to stimuli, and a character-based decoder is designed for the semantic
reconstruction of continuous language characterized by inherent character
structures. The resulting decoder can produce intelligible textual sequences
that faithfully capture the meaning of perceived speech both within and across
subjects, while existing decoders exhibit significantly inferior performance in
cross-subject contexts. The ability to decode continuous language from single
trials across subjects demonstrates the promising applications of non-invasive
language brain-computer interfaces in both healthcare and neuroscience.

通过非侵入式设备从大脑活动中解读自然语言仍然是一个巨大的挑战，本文提出了一种新颖的方法，通过三维卷积网络和信息瓶颈技术对单次非侵入性功能磁共振成像记录进行连续语言解码，该解码器能够生成捕捉到感知语音含义的可理解文本序列，并在跨受试者场景中表现出明显优于现有解码器的性能，展示了非侵入式语言脑机接口在医疗保健和神经科学领域的应用前景。

从非侵入式脑电记录中解码连续字符语言

Decoding Continuous Character-based Language from Non-invasive Brain  Recordings

Reconstructing natural videos from fMRI brain recordings is very challenging,
for two main reasons: (i) As fMRI data acquisition is difficult, we only have a
limited amount of supervised samples, which is not enough to cover the huge
space of natural videos; and (ii) The temporal resolution of fMRI recordings is
much lower than the frame rate of natural videos. In this paper, we propose a
self-supervised approach for natural-movie reconstruction. By employing
cycle-consistency over Encoding-Decoding natural videos, we can: (i) exploit
the full framerate of the training videos, and not be limited only to clips
that correspond to fMRI recordings; (ii) exploit massive amounts of external
natural videos which the subjects never saw inside the fMRI machine. These
enable increasing the applicable training data by several orders of magnitude,
introducing natural video priors to the decoding network, as well as temporal
coherence. Our approach significantly outperforms competing methods, since
those train only on the limited supervised data. We further introduce a new and
simple temporal prior of natural videos, which - when folded into our fMRI
decoder further - allows us to reconstruct videos at a higher frame-rate (HFR)
of up to x8 of the original fMRI sample rate.

本文研究自我监督方法来重建 fMRI 记录的自然视频，通过编码和解码自然视频的循环一致性以及利用大量外部自然视频，在提高适用的训练数据、引入视频先验以及保持时序一致性等方面取得了重大突破，相比于传统的只依靠有限受监督数据的方法，本文所提出的方法在性能上有了显著的提升。同时，作者还提出了一种简单的针对自然视频的时间先验，进一步提高了视频帧率。