Humans can easily deduce the relative pose of an unseen object, without
label/training, given only a single query-reference image pair. This is
arguably achieved by incorporating (i) 3D/2.5D shape perception from a single
image, (ii) render-and-compare simulation, and (iii) rich semantic cue
awareness to furnish (coarse) reference-query correspondence. Existing methods
implement (i) by a 3D CAD model or well-calibrated multiple images and (ii) by
training a network on specific objects, which necessitate laborious
ground-truth labeling and tedious training, potentially leading to challenges
in generalization. Moreover, (iii) was less exploited in the paradigm of (ii),
despite that the coarse correspondence from (iii) enhances the compare process
by filtering out non-overlapped parts under substantial pose
differences/occlusions. Motivated by this, we propose a novel 3D generalizable
relative pose estimation method by elaborating (i) with a 2.5D shape from an
RGB-D reference, (ii) with an off-the-shelf differentiable renderer, and (iii)
with semantic cues from a pretrained model like DINOv2. Specifically, our
differentiable renderer takes the 2.5D rotatable mesh textured by the RGB and
the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB
and semantic maps (with back-surface culling) under a novel rotated view. The
refinement loss comes from comparing the rendered RGB and semantic maps with
the query ones, back-propagating the gradients through the differentiable
renderer to refine the 3D relative pose. As a result, our method can be readily
applied to unseen objects, given only a single RGB-D reference, without
label/training. Extensive experiments on LineMOD, LM-O, and YCB-V show that our
training-free method significantly outperforms the SOTA supervised methods,
especially under the rigorous Acc@5/10/15{\deg} metrics and the challenging
cross-dataset settings.

用 RGB-D 参考图像的 2.5D 形状、现成可微分渲染器以及 DINOv2 预训练模型提取的语义线索为基础，无需标记和训练，提出了一种新颖的可推广的三维相对姿态估计方法，通过渲染与对比新的 RGB 和语义地图，实现了只需单个 RGB-D 参考图像可用于未见过的对象的姿态估计。