Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring
pre-trained large language models (LLMs), especially as the models' scale and
the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the
idea that the adaptation process is intrinsically low-dimensional, i.e.,
significant model changes can be represented with relatively few parameters.
However, decreasing the rank encounters challenges with generalization errors
for specific tasks when compared to full-parameter fine-tuning. We present
MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters
while maintaining a higher rank, thereby offering improved performance
potential. The core idea is to freeze original pretrained weights and train a
group of mini LoRAs with only a small number of parameters. This can capture a
significant degree of diversity among mini LoRAs, thus promoting better
generalization ability. We conduct a theoretical analysis and empirical studies
on various NLP tasks. Our experimental results show that, compared to LoRA,
MELoRA achieves better performance with 8 times fewer trainable parameters on
natural language understanding tasks and 36 times fewer trainable parameters on
instruction following tasks, which demonstrates the effectiveness of MELoRA.

MELoRA 是一种采用较少可训练参数但保持较高秩的迷你低秩适配器，用于针对预训练大型语言模型进行性能优化的方法。实验证明，在自然语言理解和指令跟随任务上，相较于低秩适配器 LoRA，MELoRA 在拥有 8 倍较少可训练参数时表现更好，而在指令跟随任务上拥有 36 倍较少可训练参数时也表现更好，证明了 MELoRA 的有效性。

用于参数高效微调的 Mini-Ensemble 低秩适配器

Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

We introduce the Sparsity Roofline, a visual performance model for evaluating
sparsity in neural networks. The Sparsity Roofline jointly models network
accuracy, sparsity, and predicted inference speedup. Our approach does not
require implementing and benchmarking optimized kernels, and the predicted
speedup is equal to what would be measured when the corresponding dense and
sparse kernels are equally well-optimized. We achieve this through a novel
analytical model for predicting sparse network performance, and validate the
predicted speedup using several real-world computer vision architectures pruned
across a range of sparsity patterns and degrees. We demonstrate the utility and
ease-of-use of our model through two case studies: (1) we show how machine
learning researchers can predict the performance of unimplemented or
unoptimized block-structured sparsity patterns, and (2) we show how hardware
designers can predict the performance implications of new sparsity patterns and
sparse data formats in hardware. In both scenarios, the Sparsity Roofline helps
performance experts identify sparsity regimes with the highest performance
potential.

我们引入了稀疏屋顶线，一种用于评估神经网络中稀疏性能的视觉性能模型。该模型同时考虑了网络准确性、稀疏性和预测的推理加速度。通过使用一种新颖的分析模型预测稀疏网络性能，并通过在多个真实世界计算机视觉架构上对稀疏模式和程度进行验证，我们验证了预测的推理加速度。我们通过两个案例研究展示了我们模型的实用性和易用性：(1) 我们展示了机器学习研究人员如何预测未实现或未优化的块结构稀疏模式的性能，(2) 我们展示了硬件设计人员如何预测硬件中新的稀疏模式和稀疏数据格式的性能影响。在这两种情况下，稀疏屋顶线帮助性能专家确定具有最高性能潜力的稀疏区域。