As the model size of pre-trained language models (PLMs) grows rapidly, full
fine-tuning becomes prohibitively expensive for model training and storage. In
vision-and-language (VL), parameter-efficient tuning (PET) techniques are
proposed to integrate modular modifications (e.g., Adapter and LoRA) into
encoder-decoder PLMs. By tuning a small set of trainable parameters, these
techniques perform on par with full fine-tuning. However, excessive modular
modifications and neglecting the functionality gap between the encoders and
decoders can lead to performance degradation, while existing PET techniques
(e.g., VL-Adapter) overlook these critical issues. In this paper, we propose a
Vision-and-Language Parameter-Efficient Tuning (VL-PET) framework to impose
effective control over modular modifications via a novel granularity-controlled
mechanism. Considering different granularity-controlled matrices generated by
this mechanism, a variety of model-agnostic VL-PET modules can be instantiated
from our framework for better efficiency and effectiveness trade-offs. We
further propose lightweight PET module designs to enhance VL alignment and
modeling for the encoders and maintain text generation for the decoders.
Extensive experiments conducted on four image-text tasks and four video-text
tasks demonstrate the efficiency, effectiveness and transferability of our
VL-PET framework. In particular, our VL-PET-large with lightweight PET module
designs significantly outperforms VL-Adapter by 2.92% (3.41%) and LoRA by 3.37%
(7.03%) with BART-base (T5-base) on image-text tasks. Furthermore, we validate
the enhanced effect of employing our VL-PET designs on existing PET techniques,
enabling them to achieve significant performance improvements. Our code is
available at this https URL

由于预训练语言模型（PLMs）的模型大小迅速增长，全面微调的训练和存储变得难以承受。在视觉与语言（VL）领域，提出了参数高效调整（PET）技术，将模块化修改（例如 Adapter 和 LoRA）整合到编码器 - 解码器 PLMs 中。通过调整一小组可训练参数，这些技术的性能与全面微调方法相当。然而，过多的模块化修改和忽视编码器和解码器之间的功能差距可能导致性能下降，而现有的 PET 技术（例如 VL-Adapter）忽视了这些关键问题。在本文中，我们提出了一种视觉与语言参数高效调整（VL-PET）框架，通过一种新的粒度控制机制对模块化修改的效果进行有效控制。考虑到此机制生成的不同粒度控制矩阵，可以从我们的框架实例化多种与模型无关的 VL-PET 模块，以获得更好的效率和效果权衡。我们进一步提出了轻量级 PET 模块设计，以增强编码器的 VL 对齐和建模能力，并保持解码器的文本生成能力。在四个图像 - 文本任务和四个视频 - 文本任务上进行的大量实验证明了我们 VL-PET 框架的效率、效果和可转移性。特别地，我们的 VL-PET-large 与轻量级 PET 模块设计在图像 - 文本任务上相比 VL-Adapter 提升了 2.92%（3.41%），相比 LoRA 提升了 3.37%（7.03%）（在 BART-base 和 T5-base 模型上）。此外，我们验证了采用我们的 VL-PET 设计对现有 PET 技术的增强效果，使其实现显著的性能提升。我们的代码可在此 https URL 获得。