Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA)
offer training efficiency on Large Language Models, but their impact on model
performance remains limited. Recent efforts integrate LoRA and
Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite
promising results, research on improving the efficiency of LoRA with MoE is
still in its early stages. Recent studies have shown that experts in the MoE
architecture have different strengths and also exhibit some redundancy. Does
this statement also apply to parameter-efficient MoE? In this paper, we
introduce a novel parameter-efficient MoE method,
\textit{\textbf{M}oE-L\textbf{o}RA with \textbf{L}ayer-wise Expert
\textbf{A}llocation (MoLA)} for Transformer-based models, where each model
layer has the flexibility to employ a varying number of LoRA experts. We
investigate several architectures with varying layer-wise expert
configurations. Experiments on six well-known NLP and commonsense QA benchmarks
demonstrate that MoLA achieves equal or superior performance compared to all
baselines. We find that allocating more LoRA experts to higher layers further
enhances the effectiveness of models with a certain number of experts in total.
With much fewer parameters, this allocation strategy outperforms the setting
with the same number of experts in every layer. This work can be widely used as
a plug-and-play parameter-efficient tuning approach for various applications.
The code is available at this https URL

研究提出了一种新颖的参数高效的 MoE 方法，称为 MoLA，适用于基于 Transformer 的模型，通过为每个模型层分配不同数量的 LoRA 专家，该方法在六个著名的 NLP 和常识 QA 基准上展示了与基线相当或更好的性能，该工作可以作为各种应用的即插即用的参数高效调优方法。