Fine-tuning large language models (LLMs) for downstream tasks can greatly
improve model quality, however serving many different fine-tuned LLMs
concurrently for users in multi-tenant environments is challenging. Dedicating
GPU memory for each model is prohibitively expensive and naively swapping large
model weights in and out of GPU memory is slow. Our key insight is that
fine-tuned models can be quickly swapped in and out of GPU memory by extracting
and compressing the delta between each model and its pre-trained base model. We
propose DeltaZip, an LLM serving system that efficiently serves multiple
full-parameter fine-tuned models concurrently by aggressively compressing model
deltas by a factor of $6\times$ to $8\times$ while maintaining high model
quality. DeltaZip increases serving throughput by $1.5\times$ to $3\times$ and
improves SLO attainment compared to a vanilla HuggingFace serving system.

通过提取和压缩模型与其预训练基础模型之间的差异，DeltaZip 是一种 LLM 服务系统，可以高效地同时提供多个全参数微调模型，压缩因子可达 6 倍至 8 倍，并且可以提高服务吞吐量 1.5 倍至 3 倍，相比于香草 HuggingFace 服务系统，可改善 SLO 达成率。