This work aims to improve the efficiency of text-to-image diffusion models.
While diffusion models use computationally expensive UNet-based denoising
operations in every generation step, we identify that not all operations are
equally relevant for the final output quality. In particular, we observe that
UNet layers operating on high-res feature maps are relatively sensitive to
small perturbations. In contrast, low-res feature maps influence the semantic
layout of the final image and can often be perturbed with no noticeable change
in the output. Based on this observation, we propose Clockwork Diffusion, a
method that periodically reuses computation from preceding denoising steps to
approximate low-res feature maps at one or more subsequent steps. For multiple
baselines, and for both text-to-image generation and image editing, we
demonstrate that Clockwork leads to comparable or improved perceptual scores
with drastically reduced computational complexity. As an example, for Stable
Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and
CLIP change.

该研究旨在提高文本到图像扩散模型的效率，通过周期性地重复利用计算以近似低分辨率特征图，在保持感知分数可比拟或改善的同时，大大减少了计算复杂度。

时钟扩散：基于模型步骤提炼的高效生成

Clockwork Diffusion: Efficient Generation With Model-Step Distillation

Perceptual distances between images, as measured in the space of pre-trained
deep features, have outperformed prior low-level, pixel-based metrics on
assessing perceptual similarity. While the capabilities of older and less
accurate models such as AlexNet and VGG to capture perceptual similarity are
well known, modern and more accurate models are less studied. In this paper, we
present a large-scale empirical study to assess how well ImageNet classifiers
perform on perceptual similarity. First, we observe a inverse correlation
between ImageNet accuracy and Perceptual Scores of modern networks such as
ResNets, EfficientNets, and Vision Transformers: that is better classifiers
achieve worse Perceptual Scores. Then, we examine the ImageNet
accuracy/Perceptual Score relationship on varying the depth, width, number of
training steps, weight decay, label smoothing, and dropout. Higher accuracy
improves Perceptual Score up to a certain point, but we uncover a Pareto
frontier between accuracies and Perceptual Score in the mid-to-high accuracy
regime. We explore this relationship further using a number of plausible
hypotheses such as distortion invariance, spatial frequency sensitivity, and
alternative perceptual functions. Interestingly we discover shallow ResNets and
ResNets trained for less than 5 epochs only on ImageNet, whose emergent
Perceptual Score matches the prior best networks trained directly on supervised
human perceptual judgements. The checkpoints for the models in our study are
available at
this https URL

该研究是一个大规模实证研究，旨在评估 ImageNet 分类器在感知相似性方面的表现。结果发现，尽管现代化的网络模型（如 ResNets、EfficientNets 和 Vision Transformers）在 ImageNet 分类方面表现良好，但它们在感知相似性上的表现较差。然而，该研究还发现，仅在 ImageNet 上训练少于 5 个时期的浅层 ResNets 和 ResNets 的感知得分，可与直接在监督人类感知判断上训练的最佳网络相匹配。