Decoding continuous language from brain activity is a formidable yet
promising field of research. It is particularly significant for aiding people
with speech disabilities to communicate through brain signals. This field
addresses the complex task of mapping brain signals to text. The previous best
attempt reverse-engineered this process in an indirect way: it began by
learning to encode brain activity from text and then guided text generation by
aligning with predicted brain responses. In contrast, we propose a simple yet
effective method that guides text reconstruction by directly comparing them
with the predicted text embeddings mapped from brain activities. Comprehensive
experiments reveal that our method significantly outperforms the current
state-of-the-art model, showing average improvements of 77% and 54% on BLEU and
METEOR scores. We further validate the proposed modules through detailed
ablation studies and case analyses and highlight a critical correlation: the
more precisely we map brain activities to text embeddings, the better the text
reconstruction results. Such insight can simplify the task of reconstructing
language from brain activities for future work, emphasizing the importance of
improving brain-to-text-embedding mapping techniques.

通过直接比较从脑活动中映射的预测文本嵌入与实际文本重建，我们的方法在文本重建方面明显胜过当前最先进模型，实验证明更精确地将脑活动映射到文本嵌入将获得更好的重建结果，这为未来的工作简化了从脑活动中重建语言的任务，强调了改进脑到文本嵌入映射技术的重要性。