Recent literature has found that an effective method to customize or further
improve large language models (LLMs) is to add dynamic adapters, such as
low-rank adapters (LoRA) with Mixture-of-Experts (MoE) structures. Though such
dynamic adapters incur modest computational complexity, they surprisingly lead
to huge inference latency overhead, slowing down the decoding speed by 2.5+
times. In this paper, we analyze the fine-grained costs of the dynamic adapters
and find that the fragmented CUDA kernel calls are the root cause. Therefore,
we propose LoRA-Switch, a system-algorithm co-designed architecture for
efficient dynamic adapters. Unlike most existing dynamic structures that adopt
layer-wise or block-wise dynamic routing, LoRA-Switch introduces a token-wise
routing mechanism. It switches the LoRA adapters and weights for each token and
merges them into the backbone for inference. For efficiency, this switching is
implemented with an optimized CUDA kernel, which fuses the merging operations
for all LoRA adapters at once. Based on experiments with popular open-source
LLMs on common benchmarks, our approach has demonstrated similar accuracy
improvement as existing dynamic adapters, while reducing the decoding latency
by more than 2.4 times.

在这篇论文中，我们提出了 LoRA-Switch，这是一种用于高效动态适配器的系统算法协同设计架构。通过在推理过程中优化了 CUDA 内核并融合了所有 LoRA 适配器的合并操作，我们的方法在保持了现有动态适配器类似准确性提升的同时，将解码延迟缩短了大于 2.4 倍。