Two fundamental questions in neurolinguistics concerns the brain regions that
integrate information beyond the lexical level, and the size of their window of
integration. To address these questions we introduce a new approach named
masked-attention generation. It uses GPT-2 transformers to generate word
embeddings that capture a fixed amount of contextual information. We then
tested whether these embeddings could predict fMRI brain activity in humans
listening to naturalistic text. The results showed that most of the cortex
within the language network is sensitive to contextual information, and that
the right hemisphere is more sensitive to longer contexts than the left.
Masked-attention generation supports previous analyses of context-sensitivity
in the brain, and complements them by quantifying the window size of context
integration per voxel.

利用 GPT-2 变形金刚生成的单词嵌入来测试大脑对自然文本中背景信息的反应，结果表明语言网络中的大多数皮层对背景信息敏感，右半球对于较长场景的反应比左半球更敏感，支持以前的上下文敏感度分析并有助于量化每个体素的上下文整合窗口大小。

使用遮蔽注意力生成探测大脑上下文敏感性

Probing Brain Context-Sensitivity with Masked-Attention Generation

A fundamental question in neurolinguistics concerns the brain regions
involved in syntactic and semantic processing during speech comprehension, both
at the lexical (word processing) and supra-lexical levels (sentence and
discourse processing). To what extent are these regions separated or
intertwined? To address this question, we trained a lexical language model,
Glove, and a supra-lexical language model, GPT-2, on a text corpus from which
we selectively removed either syntactic or semantic information. We then
assessed to what extent these information-restricted models were able to
predict the time-courses of fMRI signal of humans listening to naturalistic
text. We also manipulated the size of contextual information provided to GPT-2
in order to determine the windows of integration of brain regions involved in
supra-lexical processing. Our analyses show that, while most brain regions
involved in language are sensitive to both syntactic and semantic variables,
the relative magnitudes of these effects vary a lot across these regions.
Furthermore, we found an asymmetry between the left and right hemispheres, with
semantic and syntactic processing being more dissociated in the left hemisphere
than in the right, and the left and right hemispheres showing respectively
greater sensitivity to short and long contexts. The use of
information-restricted NLP models thus shed new light on the spatial
organization of syntactic processing, semantic processing and compositionality.

该研究使用信息限制的自然语言处理模型研究了神经语言学中与句法和语义处理有关的脑区及其空间组织，并发现左右半球在语义和句法处理方面存在差异且在上下文范围和集成窗口方面敏感性不同。