The Stable Diffusion Model (SDM) is a prevalent and effective model for
text-to-image (T2I) and image-to-image (I2I) generation. Despite various
attempts at sampler optimization, model distillation, and network
quantification, these approaches typically maintain the original network
architecture. The extensive parameter scale and substantial computational
demands have limited research into adjusting the model architecture. 1) For the
tuning method, we design a model assembly strategy to reconstruct a lightweight
model while preserving performance through distillation. Second, to mitigate
performance loss due to pruning, we incorporate multi-expert conditional
convolution (ME-CondConv) into compressed UNets to enhance network performance
by increasing capacity without sacrificing speed. Third, we validate the
effectiveness of the multi-UNet switching method for improving network speed.
2) For the tuning-free method, we propose a feature inheritance strategy to
accelerate inference by skipping local computations at the block, layer, or
unit level within the network structure. We also examine multiple sampling
modes for feature inheritance at the time-step level. Experiments demonstrate
that both the proposed tuning and the tuning-free methods can improve the speed
and performance of the SDM. The lightweight model reconstructed by the model
assembly strategy increases generation speed by $22.4\%$, while the feature
inheritance strategy enhances the SDM generation speed by $40.0\%$.

通过模型组装策略重构轻量级模型，以及通过特征继承策略加快 SDM 生成速度。

A-SDM：加速稳定扩散通过模型组装和特征继承策略

A-SDM: Accelerating Stable Diffusion through Model Assembly and Feature  Inheritance Strategies

Text-to-Image (T2I) diffusion models have achieved remarkable success in
synthesizing high-quality images conditioned on text prompts. Recent methods
have tried to replicate the success by either training text-to-video (T2V)
models on a very large number of text-video pairs or adapting T2I models on
text-video pairs independently. Although the latter is computationally less
expensive, it still takes a significant amount of time for per-video adaption.
To address this issue, we propose SAVE, a novel spectral-shift-aware adaptation
framework, in which we fine-tune the spectral shift of the parameter space
instead of the parameters themselves. Specifically, we take the spectral
decomposition of the pre-trained T2I weights and only control the change in the
corresponding singular values, i.e. spectral shift, while freezing the
corresponding singular vectors. To avoid drastic drift from the original T2I
weights, we introduce a spectral shift regularizer that confines the spectral
shift to be more restricted for large singular values and more relaxed for
small singular values. Since we are only dealing with spectral shifts, the
proposed method reduces the adaptation time significantly (approx. 10 times)
and has fewer resource constrains for training. Such attributes posit SAVE to
be more suitable for real-world applications, e.g. editing undesirable content
during video streaming. We validate the effectiveness of SAVE with an extensive
experimental evaluation under different settings, e.g. style transfer, object
replacement, privacy preservation, etc.

提出了一种新的针对文本到视频合成的迁移框架，称为 SAVE，其中将预训练模型的谱值进行微调以达到适应新任务的目的，该方法大大减少了调整模型所需的计算时间且更适合实际的视频编辑应用。