The typical process for developing LLMs involves pre-training a general
foundation model on massive data, followed by fine-tuning on task-specific data
to create specialized experts. Serving these experts poses challenges, as
loading all experts onto devices is impractical, and frequent switching between
experts in response to user requests incurs substantial I/O costs, increasing
latency and expenses. Previous approaches decompose expert weights into
pre-trained model weights and residual delta weights, then quantize the delta
weights to reduce model size. However, these methods often lead to significant
quantization errors at extremely low bitwidths and assume the appropriate model
for a user request is known in advance, which is not practical. To address
these issues, we introduce ME-Switch, a memory-efficient expert switching
framework for LLM serving. ME-Switch uses mixed-precision quantization,
selectively quantizing non-salient input channels of delta weights to extremely
low bits while keeping salient ones intact, significantly reducing storage
demands while maintaining performance. Additionally, we develop a routing
method that efficiently directs user queries to the most suitable expert by
transforming the model selection problem into a domain classification problem.
Extensive experiments show ME-Switch's promising memory efficiency and routing
performance. For example, when serving three models from the Mistral-7B family,
ME-Switch reduces model size by 1.74x while maintaining nearly lossless
performance on instruction, mathematical reasoning, and code generation tasks.
Furthermore, ME-Switch can efficiently serve 16 models from the Mistral-7B
family on a single NVIDIA A100 GPU.

ME-Switch 是一种用于 LLM 提供的高效内存专家切换框架，通过混合精度量化，选择性地对非显著输入通道进行极低位量化，同时保持显著通道完整，从而显著减少存储需求并保持性能；此外，我们还开发了一种路由方法，通过将模型选择问题转化为领域分类问题，高效地将用户查询转发到最合适的专家。广泛的实验证明 ME-Switch 在内存效率和路由性能方面具有良好的表现。