Although recent point cloud analysis achieves impressive progress, the
paradigm of representation learning from a single modality gradually meets its
bottleneck. In this work, we take a step towards more discriminative 3D point
cloud representation by fully taking advantages of images which inherently
contain richer appearance information, e.g., texture, color, and shade.
Specifically, this paper introduces a simple but effective point cloud
cross-modality training (PointCMT) strategy, which utilizes view-images, i.e.,
rendered or projected 2D images of the 3D object, to boost point cloud
analysis. In practice, to effectively acquire auxiliary knowledge from view
images, we develop a teacher-student framework and formulate the cross modal
learning as a knowledge distillation problem. PointCMT eliminates the
distribution discrepancy between different modalities through novel feature and
classifier enhancement criteria and avoids potential negative transfer
effectively. Note that PointCMT effectively improves the point-only
representation without architecture modification. Sufficient experiments verify
significant gains on various datasets using appealing backbones, i.e., equipped
with PointCMT, PointNet++ and PointMLP achieve state-of-the-art performance on
two benchmarks, i.e., 94.4% and 86.7% accuracy on ModelNet40 and ScanObjectNN,
respectively. Code will be made available at
this https URL

本文介绍了一种称为 PointCMT 的简单而有效的点云交叉模态训练策略，通过使用视图图像（即三维对象的渲染或投影的二维图像）来促进点云分析，进而更好地实现了三维点云表征。 实验证明，PointCMT 有效地改进了点云的表示方法，在不修改架构的情况下，基于 PointCMT 的模型在 ModelNet40 和 ScanObjectNN 等各种数据集上均实现了最先进的性能。