Recently, finetuning pretrained vision-language models (VLMs) has become one
prevailing paradigm to achieve state-of-the-art performance in VQA. However, as
VLMs scale, it becomes computationally expensive, storage inefficient, and
prone to overfitting to tune full model parameters for a specific task in
low-resource settings. Although current parameter-efficient tuning methods
dramatically reduce the number of tunable parameters, there still exists a
significant performance gap with full finetuning. In this paper, we propose
\textbf{MixPHM}, a redundancy-aware parameter-efficient tuning method that
outperforms full finetuning in low-resource VQA. Specifically, MixPHM is a
lightweight module implemented by multiple PHM-experts in a mixture-of-experts
manner. To reduce parameter redundancy, we reparameterize expert weights in a
low-rank subspace and share part of the weights inside and across MixPHM.
Moreover, based on our quantitative analysis of representation redundancy, we
propose \textbf{redundancy regularization}, which facilitates MixPHM to reduce
task-irrelevant redundancy while promoting task-relevant correlation.
Experiments conducted on VQA v2, GQA, and OK-VQA with different low-resource
settings show that our MixPHM outperforms state-of-the-art parameter-efficient
methods and is the only one consistently surpassing full finetuning.

本文提出了 MixPHM，这是一种重复感知的参数高效调整方法，用于在低资源 VQA 中优于完全微调。通过在混合型权威方式中实现的多个 PHM 专家来实现 MixPHM，同时重新参数化专家权重以在低秩子空间中分享权重。我们还提出了冗余正则化，促进了 MixPHM 减少任务无关冗余同时促进任务相关的相关性。在 VQA v2，GQA 和 OK-VQA 上进行的实验表明，我们的 MixPHM 优于最先进的参数高效方法，并且它是唯一一致地超越完全微调的方法。