Pre-trained computational language models have recently made remarkable progress in harnessing the language abilities which were considered unique to humans. Their success has raised interest in whether these models represent and process language like humans. To answer this question, this paper proposes MulCogBench, a multi-modal cognitive benchmark dataset collected from native Chinese and English participants. It encompasses a variety of cognitive data, including subjective semantic ratings, eye-tracking, functional magnetic resonance imaging (fMRI), and magnetoencephalography (MEG). To assess the relationship between language models and cognitive data, we conducted a similarity-encoding analysis which decodes cognitive data based on its pattern similarity with textual embeddings. Results show that language models share significant similarities with human cognitive data and the similarity patterns are modulated by the data modality and stimuli complexity. Specifically, context-aware models outperform context-independent models as language stimulus complexity increases. The shallow layers of context-aware models are better aligned with the high-temporal-resolution MEG signals whereas the deeper layers show more similarity with the high-spatial-resolution fMRI. These results indicate that language models have a delicate relationship with brain language representations. Moreover, the results between Chinese and English are highly consistent, suggesting the generalizability of these findings across languages.

本文提出了MulCogBench，这是一个从中国和英国的参与者中收集的多模态认知基准数据集，包括主观语义评分、眼动、功能性磁共振成像（fMRI）和脑磁图（MEG）等多种认知数据。结果表明，语言模型与人类认知数据有显著相似性，而相似性模式受数据模态和刺激复杂性的调节。这些结果表明语言模型与脑语言表示有微妙的关联，并且中英文之间的结果高度一致，表明这些发现具有普适性。

MulCogBench：用于评估中文和英文计算语言模型的多模态认知基准数据集