The Stable Diffusion Model (SDM) is a popular and efficient text-to-image
(t2i) generation and image-to-image (i2i) generation model. Although there have
been some attempts to reduce sampling steps, model distillation, and network
quantization, these previous methods generally retain the original network
architecture. Billion scale parameters and high computing requirements make the
research of model architecture adjustment scarce. In this work, we first
explore the computational redundancy part of the network, and then prune the
redundancy blocks of the model and maintain the network performance through a
progressive incubation strategy. Secondly, in order to maintaining the model
performance, we add cross-layer multi-expert conditional convolution
(CLME-Condconv) to the block pruning part to inherit the original convolution
parameters. Thirdly, we propose a global-regional interactive (GRI) attention
to speed up the computationally intensive attention part. Finally, we use
semantic-aware supervision (SAS) to align the outputs of the teacher model and
student model at the semantic level. Experiments show that this method can
effectively train a lightweight model close to the performance of the original
SD model, and effectively improve the model speed under limited resources.
Experiments show that the proposed method can effectively train a light-weight
model close to the performance of the original SD model, and effectively
improve the model speed under limited resources. After acceleration, the UNet
part of the model is 22% faster and the overall speed is 19% faster.

通过模型架构调整、冗余剪枝、全局 - 局部交互注意机制和语义感知监督等方法，本研究提出了一种有效训练轻量级模型、并在有限资源下提高模型速度的方法。