We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective
training to further increase parameter efficiency of the Low-rank adaptation
(LoRA) method. Our investigations include all feasible combinations parameter
training/freezing in conjunction with weight tying to identify the optimal
balance between performance and the number of trainable parameters. Through
experiments covering a variety of tasks and two base language models, we
provide analysis revealing trade-offs between efficiency and performance. Our
experiments uncovered a particular Tied-LoRA configuration that stands out by
demonstrating comparable performance across several tasks while employing only
13~\% percent of parameters utilized by the standard LoRA method.

我们提出了 Tied-LoRA，这是一个简单的范例，利用权重绑定和选择性训练来进一步提高 Low-rank 适应（LoRA）方法的参数效率。我们的研究包括将可行的参数训练 / 冻结与权重绑定相结合，以寻找性能和可训练参数数量之间的最佳平衡。通过涵盖各种任务和两个基本语言模型的实验，我们提供了分析，揭示了效率和性能之间的权衡。我们的实验发现了一种特定的 Tied-LoRA 配置，通过仅使用标准 LoRA 方法所使用参数的 13％，在几个任务上表现出可比较的性能。