Diffusion models have been shown to be capable of generating high-quality
images, suggesting that they could contain meaningful internal representations.
Unfortunately, the feature maps that encode a diffusion model's internal
information are spread not only over layers of the network, but also over
diffusion timesteps, making it challenging to extract useful descriptors. We
propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and
multi-timestep feature maps into per-pixel feature descriptors that can be used
for downstream tasks. These descriptors can be extracted for both synthetic and
real images using the generation and inversion processes. We evaluate the
utility of our Diffusion Hyperfeatures on the task of semantic keypoint
correspondence: our method achieves superior performance on the SPair-71k real
image benchmark. We also demonstrate that our method is flexible and
transferable: our feature aggregation network trained on the inversion features
of real image pairs can be used on the generation features of synthetic image
pairs with unseen objects and compositions. Our code is available at
https://diffusion-hyperfeatures.github.io.

本文介绍了 Diffusion Hyperfeatures 框架，该框架用于将多尺度和多时间步长的特征图聚合成每像素特征描述符，可以用于图像生成和反推过程，该框架在语义关键点对应任务中表现出优秀的性能，并且具有灵活的可转移性。