Exceptional text-to-image (T2I) generation results of Stable Diffusion models
(SDMs) come with substantial computational demands. To resolve this issue,
recent research on efficient SDMs has prioritized reducing the number of
sampling steps and utilizing network quantization. Orthogonal to these
directions, this study highlights the power of classical architectural
compression for general-purpose T2I synthesis by introducing block-removed
knowledge-distilled SDMs (BK-SDMs). We eliminate several residual and attention
blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of
parameters, MACs per sampling step, and latency. We conduct distillation-based
pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training
pairs) on a single A100 GPU. Despite being trained with limited resources, our
compact models can imitate the original SDM by benefiting from transferred
knowledge and achieve competitive results against larger multi-billion
parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate
the applicability of our lightweight pretrained models in personalized
generation with DreamBooth finetuning.

通过引入去块知识蒸馏稳定扩散模型（BK-SDMs）的传统架构压缩技术，从而使其可用于通用 T2I 合成，我们从 SDMs 的 U-Net 中消除了几个残差和注意块，显著地减少了参数数量、MACs 和延迟，使用少量的资源进行蒸馏式预训练，使我们的紧凑模型能够模仿原始 SDM，并在零样本 MS-COCO 基准测试中对抗更大的多十亿参数模型，同时还演示了我们轻量级预训练模型在个性化生成中的适用性。