We tackle the problem of learning an implicit scene representation for 3D
instance segmentation from a sequence of posed RGB images. Towards this, we
introduce 3DIML, a novel framework that efficiently learns a label field that
may be rendered from novel viewpoints to produce view-consistent instance
segmentation masks. 3DIML significantly improves upon training and inference
runtimes of existing implicit scene representation based methods. Opposed to
prior art that optimizes a neural field in a self-supervised manner, requiring
complicated training procedures and loss function design, 3DIML leverages a
two-phase process. The first phase, InstanceMap, takes as input 2D segmentation
masks of the image sequence generated by a frontend instance segmentation
model, and associates corresponding masks across images to 3D labels. These
almost view-consistent pseudolabel masks are then used in the second phase,
InstanceLift, to supervise the training of a neural label field, which
interpolates regions missed by InstanceMap and resolves ambiguities.
Additionally, we introduce InstanceLoc, which enables near realtime
localization of instance masks given a trained label field and an off-the-shelf
image segmentation model by fusing outputs from both. We evaluate 3DIML on
sequences from the Replica and ScanNet datasets and demonstrate 3DIML's
effectiveness under mild assumptions for the image sequences. We achieve a
14-24x speedup over existing implicit scene representation methods with
comparable quality, showcasing its potential to facilitate faster and more
effective 3D scene understanding.

我们提出了一种新的框架 3DIML，用于从一系列姿态 RGB 图像中学习隐式场景表示，以实现 3D 实例分割，该框架可以有效地学习一个标签场，并且可以在新视角中进行渲染，生成视图一致的实例分割掩码。我们通过对 Replica 和 ScanNet 数据集的序列进行评估，展示了 3DIML 在图像序列的轻度假设下的有效性，相比具有相似质量的现有方法，我们取得了 14-24 倍的加速，展示了它在加快和提高 3D 场景理解方面的潜力。