Perceiving 3D structures from RGB images based on CAD model primitives can enable an effective, efficient 3D object-based representation of scenes. However, current approaches rely on supervision from expensive annotations of CAD models associated with real images, and encounter challenges due to the inherent ambiguities in the task -- both in depth-scale ambiguity in monocular perception, as well as inexact matches of CAD database models to real observations. We thus propose DiffCAD, the first weakly-supervised probabilistic approach to CAD retrieval and alignment from an RGB image. We formulate this as a conditional generative task, leveraging diffusion to learn implicit probabilistic models capturing the shape, pose, and scale of CAD objects in an image. This enables multi-hypothesis generation of different plausible CAD reconstructions, requiring only a few hypotheses to characterize ambiguities in depth/scale and inexact shape matches. Our approach is trained only on synthetic data, leveraging monocular depth and mask estimates to enable robust zero-shot adaptation to various real target domains. Despite being trained solely on synthetic data, our multi-hypothesis approach can even surpass the supervised state-of-the-art on the Scan2CAD dataset by 5.9% with 8 hypotheses.

基于CAD模型基元从RGB图像中感知3D结构可以有效地实现基于三维物体的场景表示，本文提出了DiffCAD，这是一种第一个弱监督的条件生成方法，可以从RGB图像中检索和对齐CAD模型，通过扩散学习隐式概率模型来捕捉CAD对象在图像中的形状、姿态和尺度，实现了多个假设生成和对深度/尺度和形状匹配的歧义进行建模。

DiffCAD: 基于弱监督的概率CAD模型检索和对齐