Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large
Language Models (LLMs) with constrained resources. Although there have been
various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture
LLMs is still underexplored. In this work, we study the PEFT method for LLMs
with the Mixture-of-Experts (MoE) architecture and the contents of this work
are mainly threefold: (1) We investigate the dispersion degree of the activated
experts in customized tasks, and found that the routing distribution for a
specific task tends to be highly concentrated, while the distribution of
activated experts varies significantly across different tasks. (2) We propose
Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant
to downstream tasks while freezing the other experts and modules; experimental
results demonstrate that our method not only improves the tuning efficiency,
but also matches or even surpasses the performance of full-parameter
fine-tuning. (3) We further analyze the impact of the MoE architecture on
expert-specialized fine-tuning. We find that MoE models with finer-grained
experts are more advantageous in selecting the combination of experts that are
most relevant to downstream tasks, thereby enhancing both the training
efficiency and effectiveness.

参数高效的微调（PEFT）是为了定制具有受限资源的大型语言模型（LLMs）而关键。本文研究了 Mixture-of-Experts（MoE）架构的 LLMs 上的 PEFT 方法，主要工作内容包括：（1）研究定制任务中激活的专家的集中程度，发现特定任务的路由分布往往高度集中，而激活的专家分布在不同任务之间差异显著。（2）提出了专家定制微调（ESFT），该方法在冻结其他专家和模块的同时调整与下游任务最相关的专家；实验结果表明，我们的方法不仅提高了调整效率，还甚至超过了全参数微调的性能。（3）进一步分析 MoE 架构对专家定制微调的影响，发现具有更细粒度专家的 MoE 模型能更好地选择与下游任务最相关的专家组合，从而提高训练效率和效果。