Decoding language information from brain signals represents a vital research
area within brain-computer interfaces, particularly in the context of
deciphering the semantic information from the fMRI signal. However, many
existing efforts concentrate on decoding small vocabulary sets, leaving space
for the exploration of open vocabulary continuous text decoding. In this paper,
we introduce a novel method, the \textbf{Brain Prompt GPT (BP-GPT)}. By using
the brain representation that is extracted from the fMRI as a prompt, our
method can utilize GPT-2 to decode fMRI signals into stimulus text. Further, we
introduce a text-to-text baseline and align the fMRI prompt to the text prompt.
By introducing the text-to-text baseline, our BP-GPT can extract a more robust
brain prompt and promote the decoding of pre-trained LLM. We evaluate our
BP-GPT on the open-source auditory semantic decoding dataset and achieve a
significant improvement up to $4.61\%$ on METEOR and $2.43\%$ on BERTScore
across all the subjects compared to the state-of-the-art method. The
experimental results demonstrate that using brain representation as a prompt to
further drive LLM for auditory neural decoding is feasible and effective.

使用大脑表示作为提示进一步驱动大规模语言模型从脑信号中解码听觉神经信息是可行且有效的。