Adapting Large Language Models (LLMs) to new tasks through fine-tuning has
been made more efficient by the introduction of Parameter-Efficient Fine-Tuning
(PEFT) techniques, such as LoRA. However, these methods often underperform
compared to full fine-tuning, particularly in scenarios involving complex
datasets. This issue becomes even more pronounced in complex domains,
highlighting the need for improved PEFT approaches that can achieve better
performance. Through a series of experiments, we have uncovered two critical
insights that shed light on the training and parameter inefficiency of LoRA.
Building on these insights, we have developed HydraLoRA, a LoRA framework with
an asymmetric structure that eliminates the need for domain expertise. Our
experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even
those that rely on domain knowledge during the training and inference phases.
\href{this https URL}{Code}.

通过一系列实验，我们发现了两个关键的见解，揭示了 LoRA 的训练和参数效率问题，基于这些见解，我们开发了 HydraLoRA，这是一个具有不对称结构的 LoRA 框架，消除了对领域专业知识的需求，我们的实验表明，HydraLoRA 优于其他 PEFT 方法，即使在训练和推断阶段依赖领域知识的情况下。