We report a controlled study investigating the effect of visual information (i.e., seeing the speaker) on spoken language comprehension. We compare the ERP signature (N400) associated with each word in audio-only and audio-visual presentations of the same verbal stimuli. We assess the extent to which surprisal measures (which quantify the predictability of words in their lexical context) are generated on the basis of different types of language models (specifically n-gram and Transformer models) that predict N400 responses for each word. Our results indicate that cognitive effort differs significantly between multimodal and unimodal settings. In addition, our findings suggest that while Transformer-based models, which have access to a larger lexical context, provide a better fit in the audio-only setting, 2-gram language models are more effective in the multimodal setting. This highlights the significant impact of local lexical context on cognitive processing in a multimodal environment.

研究视觉信息（即看到发言者）对口语理解的影响，比较了同一言语刺激的仅音频和音频-视觉呈现方式下相关联的ERP签名（N400），证明在多模态和单模态情况下认知努力存在显著差异，并指出在多模态环境下局部词汇环境对认知处理的巨大影响。

音频和音视觉理解中，单词的惊奇程度是否相同？