Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI. This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal. The approach involves three stages: (1) training an EEG encoder for visual feature extraction, (2) fine-tuning LLMs on image and text data, enabling multimodal description generation, and (3) further fine-tuning on EEG embeddings to generate text directly from EEG during inference. Experiments on a public EEG dataset collected for six subjects with image stimuli demonstrate the efficacy of multimodal LLMs (LLaMa-v3, Mistral-v0.3, Qwen2.5), validated using traditional language generation evaluation metrics, GPT-4 based assessments, and evaluations by human expert. This approach marks a significant advancement towards portable, low-cost "thoughts-to-text" technology with potential applications in both neuroscience and natural language processing (NLP).

本研究解决了如何将脑活动解码为可理解文本的挑战，提出了一种名为Thought2Text的方法，利用经过脑电图数据微调的指令调优大型语言模型。实验结果显示，该方法在解释脑电图信号生成文本方面具有显著进展，展现了在神经科学和自然语言处理领域的潜在应用价值。

Thought2Text: 利用大型语言模型从脑电图信号生成文本