We present a novel 3D pose refinement approach based on differentiable
rendering for objects of arbitrary categories in the wild. In contrast to
previous methods, we make two main contributions: First, instead of comparing
real-world images and synthetic renderings in the RGB or mask space, we compare
them in a feature space optimized for 3D pose refinement. Second, we introduce
a novel differentiable renderer that learns to approximate the rasterization
backward pass from data instead of relying on a hand-crafted algorithm. For
this purpose, we predict deep cross-domain correspondences between RGB images
and 3D model renderings in the form of what we call geometric correspondence
fields. These correspondence fields serve as pixel-level gradients which are
analytically propagated backward through the rendering pipeline to perform a
gradient-based optimization directly on the 3D pose. In this way, we precisely
align 3D models to objects in RGB images which results in significantly
improved 3D pose estimates. We evaluate our approach on the challenging Pix3D
dataset and achieve up to 55% relative improvement compared to state-of-the-art
refinement methods in multiple metrics.

使用不同 iable 渲染技术，利用几何对应场将深度神经网络从 RGB 图像中学到的知识传递到 3D 坐标系中，以优化 3D 姿态估计，并成功地在 Pix3D 数据集上实现了相对于现有方法高达 55％的改进。