Text-to-image (T2I) diffusion models have demonstrated impressive image
generation capabilities. Still, their computational intensity prohibits
resource-constrained organizations from deploying T2I models after fine-tuning
them on their internal target data. While pruning techniques offer a potential
solution to reduce the computational burden of T2I models, static pruning
methods use the same pruned model for all input prompts, overlooking the
varying capacity requirements of different prompts. Dynamic pruning addresses
this issue by utilizing a separate sub-network for each prompt, but it prevents
batch parallelism on GPUs. To overcome these limitations, we introduce Adaptive
Prompt-Tailored Pruning (APTP), a novel prompt-based pruning method designed
for T2I diffusion models. Central to our approach is a prompt router model,
which learns to determine the required capacity for an input text prompt and
routes it to an architecture code, given a total desired compute budget for
prompts. Each architecture code represents a specialized model tailored to the
prompts assigned to it, and the number of codes is a hyperparameter. We train
the prompt router and architecture codes using contrastive learning, ensuring
that similar prompts are mapped to nearby codes. Further, we employ optimal
transport to prevent the codes from collapsing into a single one. We
demonstrate APTP's effectiveness by pruning Stable Diffusion (SD) V2.1 using
CC3M and COCO as target datasets. APTP outperforms the single-model pruning
baselines in terms of FID, CLIP, and CMMD scores. Our analysis of the clusters
learned by APTP reveals they are semantically meaningful. We also show that
APTP can automatically discover previously empirically found challenging
prompts for SD, e.g., prompts for generating text images, assigning them to
higher capacity codes.

T2I 扩散模型的自适应提示优化裁剪 (APTP) 通过学习提示路由模型和架构代码来减少计算负担，优于单模型裁剪，在 FID、CLIP 和 CMMD 指标上具有更好的性能，并发现语义上有意义的聚类。