This paper documents our characterization study and practices for serving
text-to-image requests with stable diffusion models in production. We first
comprehensively analyze inference request traces for commercial text-to-image
applications. It commences with our observation that add-on modules, i.e.,
ControlNets and LoRAs, that augment the base stable diffusion models, are
ubiquitous in generating images for commercial applications. Despite their
efficacy, these add-on modules incur high loading overhead, prolong the serving
latency, and swallow up expensive GPU resources. Driven by our characterization
study, we present SwiftDiffusion, a system that efficiently generates
high-quality images using stable diffusion models and add-on modules. To
achieve this, SwiftDiffusion reconstructs the existing text-to-image serving
workflow by identifying the opportunities for parallel computation and
distributing ControlNet computations across multiple GPUs. Further,
SwiftDiffusion thoroughly analyzes the dynamics of image generation and
develops techniques to eliminate the overhead associated with LoRA loading and
patching while preserving the image quality. Last, SwiftDiffusion proposes
specialized optimizations in the backbone architecture of the stable diffusion
models, which are also compatible with the efficient serving of add-on modules.
Compared to state-of-the-art text-to-image serving systems, SwiftDiffusion
reduces serving latency by up to 5x and improves serving throughput by up to 2x
without compromising image quality.

SwiftDiffusion 是一种高效生成高质量图像的系统，使用稳定的扩散模型和附加模块，通过并行计算和分布式控制网络计算，消除 LoRA 加载和合并的开销，并进行了与附加模块有效服务兼容的稳定扩散模型的优化，从而大幅降低了服务延迟，提高了服务吞吐量。

SwiftDiffusion：高效扩散模型服务与附加模块

SwiftDiffusion: Efficient Diffusion Model Serving with Add-on Modules

ControlNets are widely used for adding spatial control in image generation
with different conditions, such as depth maps, canny edges, and human poses.
However, there are several challenges when leveraging the pretrained image
ControlNets for controlled video generation. First, pretrained ControlNet
cannot be directly plugged into new backbone models due to the mismatch of
feature spaces, and the cost of training ControlNets for new backbones is a big
burden. Second, ControlNet features for different frames might not effectively
handle the temporal consistency. To address these challenges, we introduce
Ctrl-Adapter, an efficient and versatile framework that adds diverse controls
to any image/video diffusion models, by adapting pretrained ControlNets (and
improving temporal alignment for videos). Ctrl-Adapter provides diverse
capabilities including image control, video control, video control with sparse
frames, multi-condition control, compatibility with different backbones,
adaptation to unseen control conditions, and video editing. In Ctrl-Adapter, we
train adapter layers that fuse pretrained ControlNet features to different
image/video diffusion models, while keeping the parameters of the ControlNets
and the diffusion models frozen. Ctrl-Adapter consists of temporal and spatial
modules so that it can effectively handle the temporal consistency of videos.
We also propose latent skipping and inverse timestep sampling for robust
adaptation and sparse control. Moreover, Ctrl-Adapter enables control from
multiple conditions by simply taking the (weighted) average of ControlNet
outputs. With diverse image/video diffusion backbones (SDXL, Hotshot-XL,
I2VGen-XL, and SVD), Ctrl-Adapter matches ControlNet for image control and
outperforms all baselines for video control (achieving the SOTA accuracy on the
DAVIS 2017 dataset) with significantly lower computational costs (less than 10
GPU hours).

Ctrl-Adapter 是一种高效而多用途的框架，通过适应预训练的 ControlNets（并改进视频的时间对齐）来为任何图像 / 视频扩散模型添加多样的控制，包括图像控制、视频控制、稀疏帧的视频控制、多条件控制、兼容不同的骨干模型、适应不可见的控制条件和视频编辑。Ctrl-Adapter 有效地处理视频的时间一致性，通过训练适配器层，将预训练的 ControlNet 特征融合到不同的图像 / 视频扩散模型中。