The semantic understanding of indoor 3D point cloud data is crucial for a
range of subsequent applications, including indoor service robots, navigation
systems, and digital twin engineering. Global features are crucial for
achieving high-quality semantic and instance segmentation of indoor point
clouds, as they provide essential long-range context information. To this end,
we propose JSMNet, which combines a multi-layer network with a global feature
self-attention module to jointly segment three-dimensional point cloud
semantics and instances. To better express the characteristics of indoor
targets, we have designed a multi-resolution feature adaptive fusion module
that takes into account the differences in point cloud density caused by
varying scanner distances from the target. Additionally, we propose a framework
for joint semantic and instance segmentation by integrating semantic and
instance features to achieve superior results. We conduct experiments on S3DIS,
which is a large three-dimensional indoor point cloud dataset. Our proposed
method is compared against other methods, and the results show that it
outperforms existing methods in semantic and instance segmentation and provides
better results in target local area segmentation. Specifically, our proposed
method outperforms PointNet (Qi et al., 2017a) by 16.0% and 26.3% in terms of
semantic segmentation mIoU in S3DIS (Area 5) and instance segmentation mPre,
respectively. Additionally, it surpasses ASIS (Wang et al., 2019) by 6.0% and
4.6%, respectively, as well as JSPNet (Chen et al., 2022) by a margin of 3.3%
for semantic segmentation mIoU and a slight improvement of 0.3% for instance
segmentation mPre.

室内 3D 点云数据的语义理解对于室内服务机器人、导航系统和数字孪生工程等一系列后续应用非常重要。我们提出了一种名为 JSMNet 的方法，它结合了多层网络和全局特征自注意力模块，共同分割三维点云的语义和实例。通过设计了一个多分辨率特征自适应融合模块，以考虑由于传感器距目标的距离不同而导致的点云密度差异，更好地表达了室内目标的特性。此外，我们提出了一种联合语义和实例分割的框架，通过整合语义和实例特征实现更优的结果。我们在 S3DIS 上进行了实验证明，我们的方法与其他方法进行了比较，结果显示在语义和实例分割以及目标局部区域分割方面，我们的方法优于现有方法。具体而言，在 S3DIS（Area 5）的语义分割 mIoU 方面，我们的方法在 PointNet (Qi et al., 2017a) 上表现优异，提高了 16.0%，在实例分割 mPre 方面提高了 26.3%。此外，在语义分割 mIoU 方面，我们的方法超过了 ASIS (Wang et al., 2019) 6.0% 和 JPSPNet (Chen et al., 2022) 3.3%，在实例分割 mPre 方面略微提高了 0.3%。

通过自注意力和多尺度改进室内点云语义和实例分割的 JSMNet

JSMNet Improving Indoor Point Cloud Semantic and Instance Segmentation  through Self-Attention and Multiscale

Open-vocabulary scene understanding aims to localize and recognize unseen
categories beyond the annotated label space. The recent breakthrough of 2D
open-vocabulary perception is largely driven by Internet-scale paired
image-text data with rich vocabulary concepts. However, this success cannot be
directly transferred to 3D scenarios due to the inaccessibility of large-scale
3D-text pairs. To this end, we propose to distill knowledge encoded in
pre-trained vision-language (VL) foundation models through captioning
multi-view images from 3D, which allows explicitly associating 3D and
semantic-rich captions. Further, to foster coarse-to-fine visual-semantic
representation learning from captions, we design hierarchical 3D-caption pairs,
leveraging geometric constraints between 3D scenes and multi-view images.
Finally, by employing contrastive learning, the model learns language-aware
embeddings that connect 3D and text for open-vocabulary tasks. Our method not
only remarkably outperforms baseline methods by 25.8% $\sim$ 44.7% hIoU and
14.5% $\sim$ 50.4% hAP$_{50}$ in open-vocabulary semantic and instance
segmentation, but also shows robust transferability on challenging zero-shot
domain transfer tasks. See the project website at
this https URL

通过对具有语义丰富标题的多视图图像进行记录，来设计分层三维标题对，使用对比学习，学习与图像相连的语言感知嵌入，并在开放词汇语义和实例分割方面表现出卓越的性能，具有鲁棒的可迁移性。