3D open-vocabulary scene understanding aims to recognize arbitrary novel
categories beyond the base label space. However, existing works not only fail
to fully utilize all the available modal information in the 3D domain but also
lack sufficient granularity in representing the features of each modality. In
this paper, we propose a unified multimodal 3D open-vocabulary scene
understanding network, namely UniM-OV3D, which aligns point clouds with image,
language and depth. To better integrate global and local features of the point
clouds, we design a hierarchical point cloud feature extraction module that
learns comprehensive fine-grained feature representations. Further, to
facilitate the learning of coarse-to-fine point-semantic representations from
captions, we propose the utilization of hierarchical 3D caption pairs,
capitalizing on geometric constraints across various viewpoints of 3D scenes.
Extensive experimental results demonstrate the effectiveness and superiority of
our method in open-vocabulary semantic and instance segmentation, which
achieves state-of-the-art performance on both indoor and outdoor benchmarks
such as ScanNet, ScanNet200, S3IDS and nuScenes. Code is available at
this https URL

本文提出了一种统一的多模态三维开放词汇场景理解网络 (UniM-OV3D)，通过对齐点云、图像、语言和深度信息，设计了一个分层点云特征提取模块并使用了层次化三维标题对来提高粗到细点云语义表示学习，实验结果表明我们的方法在室内外基准测试中实现了开放式语义和实例分割的最先进性能。