Mainstream parameter-efficient fine-tuning (PEFT) methods, such as LoRA or Adapter, project a model's hidden states to a lower dimension, allowing pre-trained models to adapt to new data through this low-rank bottleneck. However, PEFT tasks involving multiple modalities, like vision-language (VL) tasks, require not only adaptation to new data but also learning the relationship between different modalities. Targeting at VL PEFT tasks, we propose a family of operations, called routing functions, to enhance VL alignment in the low-rank bottlenecks. The routing functions adopt linear operations and do not introduce new trainable parameters. In-depth analyses are conducted to study their behavior. In various VL PEFT settings, the routing functions significantly improve performance of the original PEFT methods, achieving over 20% improvement on VQAv2 ($\text{RoBERTa}_{\text{large}}$+ViT-L/16) and 30% on COCO Captioning (GPT2-medium+ViT-L/16). Also when fine-tuning a pre-trained multimodal model such as CLIP-BART, we observe smaller but consistent improvements across a range of VL PEFT tasks.

针对视觉-语言（VL）任务的参数高效微调方法，使用名为路由函数的操作在低秩瓶颈中增强了VL对齐，极大地改善了原始PEFT方法在各种VL PEFT设置中的表现，包括20%的提升（在VQAv2上）和30%的提升（在COCO Captioning上），同时也在多种VL PEFT任务中对预训练的多模态模型（如CLIP-BART）进行微调时观察到了较小但一致的改进。

引入路由功能以低秩瓶颈优化视觉语言参数高效微调