Parameter-efficient finetuning (PEFT) is a widely used technique to adapt
large language models for different tasks. Service providers typically create
separate systems for users to perform PEFT model finetuning and inference
tasks. This is because existing systems cannot handle workloads that include a
mix of inference and PEFT finetuning requests. As a result, shared GPU
resources are underutilized, leading to inefficiencies. To address this
problem, we present FlexLLM, the first system that can serve inference and
parameter-efficient finetuning requests in the same iteration. Our system
leverages the complementary nature of these two tasks and utilizes shared GPU
resources to run them jointly, using a method called co-serving. To achieve
this, FlexLLM introduces a novel token-level finetuning mechanism, which breaks
down the finetuning computation of a sequence into smaller token-level
computations and uses dependent parallelization and graph pruning, two static
compilation optimizations, to minimize the memory overhead and latency for
co-serving. Compared to existing systems, FlexLLM's co-serving approach reduces
the activation GPU memory overhead by up to 8x, and the end-to-end GPU memory
requirement of finetuning by up to 36% while maintaining a low inference
latency and improving finetuning throughput. For example, under a heavy
inference workload, FlexLLM can still preserve more than 80% of the peak
finetuning throughput, whereas existing systems cannot make any progress with
finetuning. The source code of FlexLLM is publicly available at
this https URL

我们提出了 FlexLLM，这是第一个能够在同一次迭代中处理推理和参数高效微调请求的系统，通过协同服务的方法，利用共享的 GPU 资源来同时运行这两个任务，FlexLLM 的合作服务方法减少了激活 GPU 内存开销高达 8 倍，并将微调的整个 GPU 内存要求降低了最多 36％，同时保持了低推理延迟并提高了微调吞吐量。