We introduce AdaMoLE, a novel method for fine-tuning large language models
(LLMs) through an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts.
Moving beyond conventional methods that employ a static top-k strategy for
activating experts, AdaMoLE dynamically adjusts the activation threshold using
a dedicated threshold network, adaptively responding to the varying
complexities of different tasks. By replacing a single LoRA in a layer with
multiple LoRA experts and integrating a gating function with the threshold
mechanism, AdaMoLE effectively selects and activates the most appropriate
experts based on the input context. Our extensive evaluations across a variety
of commonsense reasoning and natural language processing tasks show that
AdaMoLE exceeds baseline performance. This enhancement highlights the
advantages of AdaMoLE's adaptive selection of LoRA experts, improving model
effectiveness without a corresponding increase in the expert count. The
experimental validation not only confirms AdaMoLE as a robust approach for
enhancing LLMs but also suggests valuable directions for future research in
adaptive expert selection mechanisms, potentially broadening the scope for
optimizing model performance across diverse language processing tasks.

AdaMoLE 是一种通过自适应混合低秩适应（LoRA）专家的方法，用于对大型语言模型（LLMs）进行精细调节。它通过使用专用阈值网络来动态调整激活阈值，从而对不同任务的变化复杂性进行自适应响应。通过将单个 LoRA 替换为多个 LoRA 专家，并将门控函数与阈值机制集成，AdaMoLE 可以有效地选择和激活最合适的专家。通过在多种常识推理和自然语言处理任务上进行广泛评估，我们发现 AdaMoLE 的性能优于基准性能。这种改进突出了 AdaMoLE 自适应选择 LoRA 专家的优势，提高了模型的有效性而不增加专家数量。实验证实了 AdaMoLE 作为增强 LLMs 的强大方法，并且对于自适应专家选择机制的未来研究提出了有价值的方向，潜在地扩大了优化模型性能在不同语言处理任务中的范围。