Adapting Large Language Models (LLMs) to new tasks through fine-tuning has
been made more efficient by the introduction of Parameter-Efficient Fine-Tuning
(PEFT) techniques, such as LoRA. However, these methods often underperform
compared to full fine-tuning, particularly in scenarios involving complex
datasets. This issue becomes even more pronounced in complex domains,
highlighting the need for improved PEFT approaches that can achieve better
performance. Through a series of experiments, we have uncovered two critical
insights that shed light on the training and parameter inefficiency of LoRA.
Building on these insights, we have developed HydraLoRA, a LoRA framework with
an asymmetric structure that eliminates the need for domain expertise. Our
experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even
those that rely on domain knowledge during the training and inference phases.
\href{this https URL}{Code}.

通过一系列实验，我们发现了两个关键的见解，揭示了 LoRA 的训练和参数效率问题，基于这些见解，我们开发了 HydraLoRA，这是一个具有不对称结构的 LoRA 框架，消除了对领域专业知识的需求，我们的实验表明，HydraLoRA 优于其他 PEFT 方法，即使在训练和推断阶段依赖领域知识的情况下。

HydraLoRA：一种用于高效微调的非对称 LoRA 架构

HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

The entry of large language models (LLMs) into research and commercial spaces
has led to a trend of ever-larger models, with initial promises of
generalisability, followed by a widespread desire to downsize and create
specialised models without the need for complete fine-tuning, using Parameter
Efficient Fine-tuning (PEFT) methods. We present an investigation into the
suitability of different PEFT methods to clinical decision-making tasks, across
a range of model sizes, including extremely small models with as few as $25$
million parameters.
Our analysis shows that the performance of most PEFT approaches varies
significantly from one task to another, with the exception of LoRA, which
maintains relatively high performance across all model sizes and tasks,
typically approaching or matching full fine-tuned performance. The
effectiveness of PEFT methods in the clinical domain is evident, particularly
for specialised models which can operate on low-cost, in-house computing
infrastructure. The advantages of these models, in terms of speed and reduced
training costs, dramatically outweighs any performance gain from large
foundation LLMs. Furthermore, we highlight how domain-specific pre-training
interacts with PEFT methods and model size, and discuss how these factors
interplay to provide the best efficiency-performance trade-off. Full code
available at: tbd.

对不同规模的模型以及临床决策任务的适用性进行研究，揭示大型语言模型的效果与 Parameter Efficient Fine-tuning 方法的关系，发现 LoRA 方法在各项任务和模型规模下都能保持较高的性能，专用模型在速度和训练成本上具有优越性，与大型基础语言模型相比效果更好，同时探讨了领域特定预训练与 PEFT 方法和模型规模之间的相互影响，以及提供最佳效率与性能平衡的因素。