The advanced language processing abilities of large language models (LLMs)
have stimulated debate over their capacity to replicate human-like cognitive
processes. One differentiating factor between language processing in LLMs and
humans is that language input is often grounded in more than one perceptual
modality, whereas most LLMs process solely text-based information. Multimodal
grounding allows humans to integrate - e.g. visual context with linguistic
information and thereby place constraints on the space of upcoming words,
reducing cognitive load and improving perception and comprehension. Recent
multimodal LLMs (mLLMs) combine visual and linguistic embedding spaces with a
transformer type attention mechanism for next-word prediction. To what extent
does predictive language processing based on multimodal input align in mLLMs
and humans? To answer this question, 200 human participants watched short
audio-visual clips and estimated the predictability of an upcoming verb or
noun. The same clips were processed by the mLLM CLIP, with predictability
scores based on a comparison of image and text feature vectors. Eye-tracking
was used to estimate what visual features participants attended to, and CLIP's
visual attention weights were recorded. We find that human estimates of
predictability align significantly with CLIP scores, but not for a unimodal LLM
of comparable parameter size. Further, alignment vanished when CLIP's visual
attention weights were perturbed, and when the same input was fed to a
multimodal model without attention. Analysing attention patterns, we find a
significant spatial overlap between CLIP's visual attention weights and human
eye-tracking data. Results suggest that comparable processes of integrating
multimodal information, guided by attention to relevant visual features,
supports predictive language processing in mLLMs and humans.

大型语言模型（LLMs）的高级语言处理能力引发了关于它们是否能够复制类似人类认知过程的能力的讨论，本文通过研究多模态语言模型（mLLMs）中的视觉关注权重，发现与人类一样，mLLMs 中基于多模态输入的预测性语言处理过程也会受到视觉特征的注意引导。