Vision-Language models (VLMs) pre-trained on large corpora have demonstrated notable success across a range of downstream tasks. In light of the rapidly increasing size of pre-trained VLMs, parameter-efficient transfer learning (PETL) has garnered attention as a viable alternative to full fine-tuning. One such approach is the adapter, which introduces a few trainable parameters into the pre-trained models while preserving the original parameters during adaptation. In this paper, we present a novel modeling framework that recasts adapter tuning after attention as a graph message passing process on attention graphs, where the projected query and value features and attention matrix constitute the node features and the graph adjacency matrix, respectively. Within this framework, tuning adapters in VLMs necessitates handling heterophilic graphs, owing to the disparity between the projected query and value space. To address this challenge, we propose a new adapter architecture, $p$-adapter, which employs $p$-Laplacian message passing in Graph Neural Networks (GNNs). Specifically, the attention weights are re-normalized based on the features, and the features are then aggregated using the calibrated attention matrix, enabling the dynamic exploitation of information with varying frequencies in the heterophilic attention graphs. We conduct extensive experiments on different pre-trained VLMs and multi-modal tasks, including visual question answering, visual entailment, and image captioning. The experimental results validate our method's significant superiority over other PETL methods.

本文提出了一种新的建模框架，将适配器调谐视觉-语言模型中的注意力转化为基于注意力图的图消息传递过程，其中，投影查询和值特征以及注意力矩阵构成节点特征和图邻接矩阵。在这个框架中，适配器调谐视觉-语言模型需要处理异种图，为此，我们提出了一种新的适配器架构$p$-adapter，在图神经网络中使用$p$-Laplacian消息传递。我们在不同的预训练的视觉-语言模型和多模态任务上进行了大量实验，包括视觉问答、视觉蕴涵和图像字幕生成。实验结果验证了我们的方法在参数高效的迁移学习方法中的显著优势。

基于p-Laplacian的适应性生成预训练视觉-语言模型