Decoding non-invasive cognitive signals to natural language has long been the
goal of building practical brain-computer interfaces (BCIs). Recent major
milestones have successfully decoded cognitive signals like functional Magnetic
Resonance Imaging (fMRI) and electroencephalogram (EEG) into text under open
vocabulary setting. However, how to split the datasets for training,
validating, and testing in cognitive signal decoding task still remains
controversial. In this paper, we conduct systematic analysis on current dataset
splitting methods and find the existence of data contamination largely
exaggerates model performance. Specifically, first we find the leakage of test
subjects' cognitive signals corrupts the training of a robust encoder. Second,
we prove the leakage of text stimuli causes the auto-regressive decoder to
memorize information in test set. The decoder generates highly accurate text
not because it truly understands cognitive signals. To eliminate the influence
of data contamination and fairly evaluate different models' generalization
ability, we propose a new splitting method for different types of cognitive
datasets (e.g. fMRI, EEG). We also test the performance of SOTA Brain-to-Text
decoding models under the proposed dataset splitting paradigm as baselines for
further research.

非侵入性的认知信号解码为自然语言一直是建立实用脑 - 计算机接口（BCIs）的目标。最近的重要里程碑成功地将认知信号如功能磁共振成像（fMRI）和脑电图（EEG）在开放词汇设置下解码为文本。然而，如何在认知信号解码任务中拆分训练、验证和测试数据集仍然存在争议。本文对当前数据集拆分方法进行了系统分析，发现数据污染的存在大大夸大了模型性能。为了消除数据污染的影响和公平评估不同模型的泛化能力，我们提出了一种新的分裂方法用于不同类型的认知数据集（例如 fMRI，EEG）。我们还测试了在提出的数据集分裂模式下的 SOTA 脑 - 文本解码模型的性能作为后续研究的基准。