We use (multi)modal deep neural networks (DNNs) to probe for sites of
multimodal integration in the human brain by predicting stereoencephalography
(SEEG) recordings taken while human subjects watched movies. We operationalize
sites of multimodal integration as regions where a multimodal vision-language
model predicts recordings better than unimodal language, unimodal vision, or
linearly-integrated language-vision models. Our target DNN models span
different architectures (e.g., convolutional networks and transformers) and
multimodal training techniques (e.g., cross-attention and contrastive
learning). As a key enabling step, we first demonstrate that trained vision and
language models systematically outperform their randomly initialized
counterparts in their ability to predict SEEG signals. We then compare unimodal
and multimodal models against one another. Because our target DNN models often
have different architectures, number of parameters, and training sets (possibly
obscuring those differences attributable to integration), we carry out a
controlled comparison of two models (SLIP and SimCLR), which keep all of these
attributes the same aside from input modality. Using this approach, we identify
a sizable number of neural sites (on average 141 out of 1090 total sites or
12.94%) and brain regions where multimodal integration seems to occur.
Additionally, we find that among the variants of multimodal training techniques
we assess, CLIP-style training is the best suited for downstream prediction of
the neural activity in these sites.

使用多模态深度神经网络（DNN）预测了人脑多模态整合的位置，通过预测人类在观看电影时的立体脑电图（SEEG）记录，我们将多模态整合的位置定义为多模态视觉语言模型优于单模态语言、单模态视觉或线性整合的区域。通过对不同架构和多模态训练技术的目标 DNN 模型进行对比，我们发现在大脑区域中存在多个整合多模态信息的神经位点。此外，我们发现在我们评估的多模态训练技术变体中，CLIP 风格的训练最适合下游预测这些位点的神经活动。