Online construction of open-ended language scenes is crucial for robotic
applications, where open-vocabulary interactive scene understanding is
required. Recently, neural implicit representation has provided a promising
direction for online interactive mapping. However, implementing open-vocabulary
scene understanding capability into online neural implicit mapping still faces
three challenges: lack of local scene updating ability, blurry spatial
hierarchical semantic segmentation and difficulty in maintaining multi-view
consistency. To this end, we proposed O2V-mapping, which utilizes voxel-based
language and geometric features to create an open-vocabulary field, thus
allowing for local updates during online training process. Additionally, we
leverage a foundational model for image segmentation to extract language
features on object-level entities, achieving clear segmentation boundaries and
hierarchical semantic features. For the purpose of preserving consistency in 3D
object properties across different viewpoints, we propose a spatial adaptive
voxel adjustment mechanism and a multi-view weight selection method. Extensive
experiments on open-vocabulary object localization and semantic segmentation
demonstrate that O2V-mapping achieves online construction of language scenes
while enhancing accuracy, outperforming the previous SOTA method.

O2V-mapping utilizes voxel-based language and geometric features to achieve online construction of open-vocabulary language scenes, overcoming challenges such as lack of local scene updating ability, blurry spatial hierarchical semantic segmentation, and difficulty in maintaining multi-view consistency.

O2V-Mapping：基于神经隐性表示的在线开放词汇映射

O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit  Representation

Open-vocabulary scene understanding aims to localize and recognize unseen
categories beyond the annotated label space. The recent breakthrough of 2D
open-vocabulary perception is largely driven by Internet-scale paired
image-text data with rich vocabulary concepts. However, this success cannot be
directly transferred to 3D scenarios due to the inaccessibility of large-scale
3D-text pairs. To this end, we propose to distill knowledge encoded in
pre-trained vision-language (VL) foundation models through captioning
multi-view images from 3D, which allows explicitly associating 3D and
semantic-rich captions. Further, to foster coarse-to-fine visual-semantic
representation learning from captions, we design hierarchical 3D-caption pairs,
leveraging geometric constraints between 3D scenes and multi-view images.
Finally, by employing contrastive learning, the model learns language-aware
embeddings that connect 3D and text for open-vocabulary tasks. Our method not
only remarkably outperforms baseline methods by 25.8% $\sim$ 44.7% hIoU and
14.5% $\sim$ 50.4% hAP$_{50}$ in open-vocabulary semantic and instance
segmentation, but also shows robust transferability on challenging zero-shot
domain transfer tasks. See the project website at
this https URL

通过对具有语义丰富标题的多视图图像进行记录，来设计分层三维标题对，使用对比学习，学习与图像相连的语言感知嵌入，并在开放词汇语义和实例分割方面表现出卓越的性能，具有鲁棒的可迁移性。