Recent advancements in Large Language Models (LLMs) have achieved robust performance across diverse tasks, but fine-tuning these models for specific domains remains resource-intensive. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) address this challenge by fine-tuning a small subset of parameters. However, existing methods for fusing multiple LoRAs lack dynamic fusion based on contextual inputs and often increase inference time due to token-level operations. We propose DLP-LoRA, a Dynamic Lightweight Plugin that employs a mini-MLP module with only 5M parameters to dynamically fuse multiple LoRAs at the sentence level using top-p sampling strategies. This approach reduces inference time to less than twice that of single LoRA inference by leveraging parallel computation. Evaluations across 26 tasks-including multiple-choice questions and question answering-demonstrate that DLP-LoRA achieves an average accuracy of 92.34% on multiple-choice datasets and significant improvements in BLEU and ROUGE scores on QA datasets, outperforming different LLMs backbones under composite task settings. DLP-LoRA effectively balances performance and efficiency, making it a practical solution for dynamic multi-task adaptation in LLMs. Our code is available at https://github.com/MeCuping/DLP-LoRA.

本研究解决了大型语言模型（LLMs）在特定领域微调时高资源消耗的问题，提出了一种动态轻量级插件DLP-LoRA，通过使用仅5M参数的小型MLP模块，在句子层面动态融合多个LoRA，提高了推理效率。实验结果显示，DLP-LoRA在多个任务上表现优异，平均准确率达92.34%，并在问答数据集上显著提高了BLEU和ROUGE分数，展现出高效的多任务适应能力。

DLP-LoRA：用于大型语言模型的动态轻量级插件以实现高效任务特定LoRA融合