We present Diff3F as a simple, robust, and class-agnostic feature descriptor
that can be computed for untextured input shapes (meshes or point clouds). Our
method distills diffusion features from image foundational models onto input
shapes. Specifically, we use the input shapes to produce depth and normal maps
as guidance for conditional image synthesis, and in the process produce
(diffusion) features in 2D that we subsequently lift and aggregate on the
original surface. Our key observation is that even if the conditional image
generations obtained from multi-view rendering of the input shapes are
inconsistent, the associated image features are robust and can be directly
aggregated across views. This produces semantic features on the input shapes,
without requiring additional data or training. We perform extensive experiments
on multiple benchmarks (SHREC'19, SHREC'20, and TOSCA) and demonstrate that our
features, being semantic instead of geometric, produce reliable correspondence
across both isometeric and non-isometrically related shape families.

我们提出了 Diff3F 作为一种简单、健壮且与类别无关的特征描述符，可用于计算无纹理输入形状（网格或点云）。我们的方法从图像基础模型中提取扩散特征到输入形状。通过使用输入形状生成深度和法线图作为条件图像合成的指导，我们能够在 2D 上生成（扩散）特征，并在原始表面上进行提升和聚合。我们的关键观察是，即使从输入形状的多视图渲染得到的条件图像生成结果不一致，相关的图像特征是稳健的，并且可以直接在视图间进行聚合。这产生了输入形状上的语义特征，而无需额外的数据或训练。我们在多个基准测试（SHREC'19、SHREC'20 和 TOSCA）上进行了大量实验证明，我们的特征，作为语义而不是几何特征，能够在同构和非同构相关的形状族之间产生可靠的对应关系。

Diffusion 3D 特征（Diff3F）：用精炼的语义特征装饰无纹理形状

Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with  Distilled Semantic Features

Matching cross-modality features between images and point clouds is a
fundamental problem for image-to-point cloud registration. However, due to the
modality difference between images and points, it is difficult to learn robust
and discriminative cross-modality features by existing metric learning methods
for feature matching. Instead of applying metric learning on cross-modality
data, we propose to unify the modality between images and point clouds by
pretrained large-scale models first, and then establish robust correspondence
within the same modality. We show that the intermediate features, called
diffusion features, extracted by depth-to-image diffusion models are
semantically consistent between images and point clouds, which enables the
building of coarse but robust cross-modality correspondences. We further
extract geometric features on depth maps produced by the monocular depth
estimator. By matching such geometric features, we significantly improve the
accuracy of the coarse correspondences produced by diffusion features.
Extensive experiments demonstrate that without any task-specific training,
direct utilization of both features produces accurate image-to-point cloud
registration. On three public indoor and outdoor benchmarks, the proposed
method averagely achieves a 20.6 percent improvement in Inlier Ratio, a
three-fold higher Inlier Number, and a 48.6 percent improvement in Registration
Recall than existing state-of-the-arts.

图像与点云之间的特征匹配问题是图像到点云配准的基本问题。通过预训练的大规模模型来统一图像和点云之间的模态，并在同一模态内建立可靠的对应关系。我们利用深度到图像扩散模型提取的中间特征即扩散特征在图像和点云之间具有语义一致性，这使得粗糙但稳健的跨模态对应关系的建立成为可能。在由单目深度估计器产生的深度图上进一步提取几何特征，通过匹配这些几何特征，显著提高了扩散特征产生的粗糙对应关系的准确性。在三个公共室内外基准测试上，通过直接利用这两种特征，该方法平均提高了 20.6％的内点比率、三倍的内点数目和 48.6％的配准召回率相比现有最先进方法。

FreeReg: 集成预训练扩散模型和单目深度估计的图像到点云注册

FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained  Diffusion Models and Monocular Depth Estimators

Finding correspondences between images is a fundamental problem in computer
vision. In this paper, we show that correspondence emerges in image diffusion
models without any explicit supervision. We propose a simple strategy to
extract this implicit knowledge out of diffusion networks as image features,
namely DIffusion FeaTures (DIFT), and use them to establish correspondences
between real images. Without any additional fine-tuning or supervision on the
task-specific data or annotations, DIFT is able to outperform both
weakly-supervised methods and competitive off-the-shelf features in identifying
semantic, geometric, and temporal correspondences. Particularly for semantic
correspondence, DIFT from Stable Diffusion is able to outperform DINO and
OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k
benchmark. It even outperforms the state-of-the-art supervised methods on 9 out
of 18 categories while remaining on par for the overall performance. Project
page: this https URL

本文提出 DIffusion FeaTures (DIFT) 方法来确立图像之间的对应关系，在稳定扩散的情况下，DIFT 均能够在语义、几何和时间上优于弱监督方法以及其他对照组，尤其是在 SPair-71k 基准测试中，DIFT 相对于 DINO 和 OpenCLIP 的准确率分别提高了 19 个和 14 个点，即使在总体性能上表现相当的情况下，在 18 个类别中有 9 个可以超越最先进的监督方法。