In this work, we address the challenging task of few-shot and zero-shot 3D
point cloud semantic segmentation. The success of few-shot semantic
segmentation in 2D computer vision is mainly driven by the pre-training on
large-scale datasets like imagenet. The feature extractor pre-trained on
large-scale 2D datasets greatly helps the 2D few-shot learning. However, the
development of 3D deep learning is hindered by the limited volume and instance
modality of datasets due to the significant cost of 3D data collection and
annotation. This results in less representative features and large intra-class
feature variation for few-shot 3D point cloud segmentation. As a consequence,
directly extending existing popular prototypical methods of 2D few-shot
classification/segmentation into 3D point cloud segmentation won't work as well
as in 2D domain. To address this issue, we propose a Query-Guided Prototype
Adaption (QGPA) module to adapt the prototype from support point clouds feature
space to query point clouds feature space. With such prototype adaption, we
greatly alleviate the issue of large feature intra-class variation in point
cloud and significantly improve the performance of few-shot 3D segmentation.
Besides, to enhance the representation of prototypes, we introduce a
Self-Reconstruction (SR) module that enables prototype to reconstruct the
support mask as well as possible. Moreover, we further consider zero-shot 3D
point cloud semantic segmentation where there is no support sample. To this
end, we introduce category words as semantic information and propose a
semantic-visual projection model to bridge the semantic and visual spaces. Our
proposed method surpasses state-of-the-art algorithms by a considerable 7.90%
and 14.82% under the 2-way 1-shot setting on S3DIS and ScanNet benchmarks,
respectively. Code is available at this https URL

本文提出了 Query-Guided Prototype Adaption (QGPA) 模块并通过引入 Self-Reconstruction （SR）来增强原型的表示以适应点云的大内部类特征差异和极少量数据，最终在 S3DIS 和 ScanNet 基准测试中超越最先进的算法，其中还考虑到零样本的情况