Latent Diffusion Models (LDMs) have emerged as powerful generative models,
known for delivering remarkable results under constrained computational
resources. However, deploying LDMs on resource-limited devices remains a
complex issue, presenting challenges such as memory consumption and inference
speed. To address this issue, we introduce LD-Pruner, a novel
performance-preserving structured pruning method for compressing LDMs.
Traditional pruning methods for deep neural networks are not tailored to the
unique characteristics of LDMs, such as the high computational cost of training
and the absence of a fast, straightforward and task-agnostic method for
evaluating model performance. Our method tackles these challenges by leveraging
the latent space during the pruning process, enabling us to effectively
quantify the impact of pruning on model performance, independently of the task
at hand. This targeted pruning of components with minimal impact on the output
allows for faster convergence during training, as the model has less
information to re-learn, thereby addressing the high computational cost of
training. Consequently, our approach achieves a compressed model that offers
improved inference speed and reduced parameter count, while maintaining minimal
performance degradation. We demonstrate the effectiveness of our approach on
three different tasks: text-to-image (T2I) generation, Unconditional Image
Generation (UIG) and Unconditional Audio Generation (UAG). Notably, we reduce
the inference time of Stable Diffusion (SD) by 34.9% while simultaneously
improving its FID by 5.2% on MS-COCO T2I benchmark. This work paves the way for
more efficient pruning methods for LDMs, enhancing their applicability.

通过利用潜在空间进行剪枝过程，LD-Pruner 方法旨在提供用于压缩 LDMs 的有效性评估模型性能的方法，从而减少训练的高计算成本和提高推理速度。