The Stable Diffusion Model (SDM) is a prevalent and effective model for
text-to-image (T2I) and image-to-image (I2I) generation. Despite various
attempts at sampler optimization, model distillation, and network
quantification, these approaches typically maintain the original network
architecture. The extensive parameter scale and substantial computational
demands have limited research into adjusting the model architecture. 1) For the
tuning method, we design a model assembly strategy to reconstruct a lightweight
model while preserving performance through distillation. Second, to mitigate
performance loss due to pruning, we incorporate multi-expert conditional
convolution (ME-CondConv) into compressed UNets to enhance network performance
by increasing capacity without sacrificing speed. Third, we validate the
effectiveness of the multi-UNet switching method for improving network speed.
2) For the tuning-free method, we propose a feature inheritance strategy to
accelerate inference by skipping local computations at the block, layer, or
unit level within the network structure. We also examine multiple sampling
modes for feature inheritance at the time-step level. Experiments demonstrate
that both the proposed tuning and the tuning-free methods can improve the speed
and performance of the SDM. The lightweight model reconstructed by the model
assembly strategy increases generation speed by $22.4\%$, while the feature
inheritance strategy enhances the SDM generation speed by $40.0\%$.

通过模型组装策略重构轻量级模型，以及通过特征继承策略加快 SDM 生成速度。