Text-to-image diffusion models have made significant advances in generating
and editing high-quality images. As a result, numerous approaches have explored
the ability of diffusion model features to understand and process single images
for downstream tasks, e.g., classification, semantic segmentation, and
stylization. However, significantly less is known about what these features
reveal across multiple, different images and objects. In this work, we exploit
Stable Diffusion (SD) features for semantic and dense correspondence and
discover that with simple post-processing, SD features can perform
quantitatively similar to SOTA representations. Interestingly, the qualitative
analysis reveals that SD features have very different properties compared to
existing representation learning features, such as the recently released
DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide
high-quality spatial information but sometimes inaccurate semantic matches. We
demonstrate that a simple fusion of these two features works surprisingly well,
and a zero-shot evaluation using nearest neighbors on these fused features
provides a significant performance gain over state-of-the-art methods on
benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS. We also show that
these correspondences can enable interesting applications such as instance
swapping in two images.

该研究介绍了如何使用 Stable Diffusion 特征进行语义和密集的图像特征对应，并发现这些特征与现有的特征学习特征有很大的不同。他们提出了一种简单的融合方案，利用融合特征进行零样本最近邻方法，此方法在 SPair-71k、PF-Pascal、TSS 等标准数据集上的表现超过同类技术，同时还展示了特征对应可以实现有趣的应用，如两张图片的实例置换。