This paper presents BELT, a novel model and learning framework for the
pivotal topic of brain-to-language translation research. The translation from
noninvasive brain signals into readable natural language has the potential to
promote the application scenario as well as the development of brain-computer
interfaces (BCI) as a whole. The critical problem in brain signal decoding or
brain-to-language translation is the acquisition of semantically appropriate
and discriminative EEG representation from a dataset of limited scale and
quality. The proposed BELT method is a generic and efficient framework that
bootstraps EEG representation learning using off-the-shelf large-scale
pretrained language models (LMs). With a large LM's capacity for understanding
semantic information and zero-shot generalization, BELT utilizes large LMs
trained on Internet-scale datasets to bring significant improvements to the
understanding of EEG signals.
In particular, the BELT model is composed of a deep conformer encoder and a
vector quantization encoder. Semantical EEG representation is achieved by a
contrastive learning step that provides natural language supervision. We
achieve state-of-the-art results on two featuring brain decoding tasks
including the brain-to-language translation and zero-shot sentiment
classification. Specifically, our model surpasses the baseline model on both
tasks by 5.45% and over 10% and archives a 42.31% BLEU-1 score and 67.32%
precision on the main evaluation metrics for translation and zero-shot
sentiment classification respectively.

提出一种名为 BELT 的模型和学习框架，以大规模预训练语言模型和自然语言监督为基础，通过深度 Conformer 编码器和矢量量化编码器实现大脑信号到自然语言的转换，取得了在脑信号解码和零样本情感分类任务上的最新结果。