This paper introduces a novel Parameter-Efficient Fine-Tuning (PEFT)
framework for multi-modal, multi-task transfer learning with pre-trained
language models. PEFT techniques such as LoRA, BitFit and IA3 have demonstrated
comparable performance to full fine-tuning of pre-trained models for specific
downstream tasks, all while demanding significantly fewer trainable parameters
and reduced GPU memory consumption. However, in the context of multi-modal
fine-tuning, the need for architectural modifications or full fine-tuning often
becomes apparent. To address this we propose Context-PEFT, which learns
different groups of adaptor parameters based on the token's domain. This
approach enables LoRA-like weight injection without requiring additional
architectural changes. Our method is evaluated on the COCO captioning task,
where it outperforms full fine-tuning under similar data constraints while
simultaneously offering a substantially more parameter-efficient and
computationally economical solution.

提出了一种用于多模态、多任务迁移学习的新型参数高效调参方法（PEFT）框架，它通过 LoRA、BitFit 和 IA3 等技术，在几乎不需要可训练参数和 GPU 内存的情况下，展示了与预训练模型完全微调相当的性能，然而，在多模态微调中，经常需要进行架构修改或完全微调。为了解决这个问题，我们提出了 Context-PEFT，它根据令牌的领域学习不同的适配器参数组，这种方法使得可以实现类似 LoRA 的权重注入，而不需要额外的架构修改。我们的方法在 COCO 字幕任务上进行评估，在类似的数据限制下，优于完全微调，并同时提供了更高的参数效率和计算经济性的解决方案。

Context-PEFT: 高效多模态、多任务微调

Context-PEFT: Efficient Multi-Modal, Multi-Task Fine-Tuning

Recent studies have demonstrated Large Language Models (LLMs) can extend
their zero-shot generalization capabilities to multimodal learning through
instruction tuning. As more modalities and downstream tasks are introduced,
negative conflicts and interference may have a worse impact on performance.
While this phenomenon has been overlooked in previous work, we propose a novel
and extensible framework, called \mname, for comprehensive studies and
experimentation on multimodal learning with Multimodal Large Language Models
(MLLMs). Specifically, we combine the well-known Mixture-of-Experts (MoE) and
one of the representative PEFT techniques, \emph{i.e.,} LoRA, designing a novel
LLM-based decoder, called LoRA-MoE, for multimodal learning. The experimental
results (about 20\% improvement) have shown the effectiveness and versatility
of our design in various 2D and 3D downstream tasks. Code and corresponding
dataset will be available soon.

通过介绍新颖且可扩展的框架 LaRA-MoE，本研究通过大型语言模型在多模态学习方面的综合研究和实验，展示了在各种二维和三维下游任务中设计的开创性的 LoRA-MoE 解码器的有效性和多样性（提升约 20%），以及更多其他模态和任务的导入在性能上可能引起的负面冲突和干扰。