In this paper, we propose a novel joint instance and semantic segmentation
approach, which is called JSNet, in order to address the instance and semantic
segmentation of 3D point clouds simultaneously. Firstly, we build an effective
backbone network to extract robust features from the raw point clouds.
Secondly, to obtain more discriminative features, a point cloud feature fusion
module is proposed to fuse the different layer features of the backbone
network. Furthermore, a joint instance semantic segmentation module is
developed to transform semantic features into instance embedding space, and
then the transformed features are further fused with instance features to
facilitate instance segmentation. Meanwhile, this module also aggregates
instance features into semantic feature space to promote semantic segmentation.
Finally, the instance predictions are generated by applying a simple mean-shift
clustering on instance embeddings. As a result, we evaluate the proposed JSNet
on a large-scale 3D indoor point cloud dataset S3DIS and a part dataset
ShapeNet, and compare it with existing approaches. Experimental results
demonstrate our approach outperforms the state-of-the-art method in 3D instance
segmentation with a significant improvement in 3D semantic prediction and our
method is also beneficial for part segmentation. The source code for this work
is available at this https URL

提出了一种新的联合实例和语义分割方法，称为 JSNet，以同时处理 3D 点云的实例和语义分割。该方法包括构建一个有效的主干网络，通过点云特征融合模块融合主干网络的不同层特征以及开发联合实例语义分割模块，把语义特征转换成实例嵌入空间并将其与实例特征相结合以促进实例分割，并将实例特征聚合到语义特征空间以促进语义分割，最后采用简单的均值漂移聚类生成实例预测。实验结果表明，提出的 JSNet 在 3D 实例分割方面比现有方法表现更好，在 3D 语义预测方面也有显著改善，并且还有益于部分分割。