This paper proposes a cross-modal distillation framework, PartDistill, which transfers 2D knowledge from vision-language models (VLMs) to facilitate 3D shape part segmentation. PartDistill addresses three major challenges in this task: the lack of 3D segmentation in invisible or undetected regions in the 2D projections, inaccurate and inconsistent 2D predictions by VLMs, and the lack of knowledge accumulation across different 3D shapes. PartDistill consists of a teacher network that uses a VLM to make 2D predictions and a student network that learns from the 2D predictions while extracting geometrical features from multiple 3D shapes to carry out 3D part segmentation. A bi-directional distillation, including forward and backward distillations, is carried out within the framework, where the former forward distills the 2D predictions to the student network, and the latter improves the quality of the 2D predictions, which subsequently enhances the final 3D part segmentation. Moreover, PartDistill can exploit generative models that facilitate effortless 3D shape creation for generating knowledge sources to be distilled. Through extensive experiments, PartDistill boosts the existing methods with substantial margins on widely used ShapeNetPart and PartE datasets, by more than 15% and 12% higher mIoU scores, respectively.

该研究提出了一个跨模态蒸馏框架PartDistill，将二维视觉-语言模型的知识转移给三维形状的部分分割，通过解决三个主要挑战（二维投影中不可见或未检测区域的三维分割缺失、VLM的二维预测不准确和不一致、不同三维形状之间的知识积累不足），PartDistill在该任务中取得了显著的进展。通过在框架内进行正向和反向的蒸馏，PartDistill能够提高最终的三维部分分割结果，并且可以利用生成模型生成用于蒸馏的知识源。在广泛使用的ShapeNetPart和PartE数据集上进行的大量实验证明，PartDistill相比现有方法在mIoU分数上显著提升，分别高出15%和12%。

PartDistill: 视觉语言模型蒸馏下的三维形状部分分割