Cross-modal knowledge transfer enhances point cloud representation learning
in LiDAR semantic segmentation. Despite its potential, the \textit{weak teacher
challenge} arises due to repetitive and non-diverse car camera images and
sparse, inaccurate ground truth labels. To address this, we propose the
Efficient Image-to-LiDAR Knowledge Transfer (ELiTe) paradigm. ELiTe introduces
Patch-to-Point Multi-Stage Knowledge Distillation, transferring comprehensive
knowledge from the Vision Foundation Model (VFM), extensively trained on
diverse open-world images. This enables effective knowledge transfer to a
lightweight student model across modalities. ELiTe employs Parameter-Efficient
Fine-Tuning to strengthen the VFM teacher and expedite large-scale model
training with minimal costs. Additionally, we introduce the Segment Anything
Model based Pseudo-Label Generation approach to enhance low-quality image
labels, facilitating robust semantic representations. Efficient knowledge
transfer in ELiTe yields state-of-the-art results on the SemanticKITTI
benchmark, outperforming real-time inference models. Our approach achieves this
with significantly fewer parameters, confirming its effectiveness and
efficiency.

通过跨模态知识转移增强了点云表示学习在激光雷达语义分割中的应用，引入有效的知识转移策略和众多优化方法，得到了领先的性能并且具有较低的模型参数。

ELiTe: 高效的图像到 LiDAR 的知识转移用于语义分割

ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic  Segmentation

In perception, multiple sensory information is integrated to map visual
information from 2D views onto 3D objects, which is beneficial for
understanding in 3D environments. But in terms of a single 2D view rendered
from different angles, only limited partial information can be provided.The
richness and value of Multi-view 2D information can provide superior
self-supervised signals for 3D objects. In this paper, we propose a novel
self-supervised point cloud representation learning method, MM-Point, which is
driven by intra-modal and inter-modal similarity objectives. The core of
MM-Point lies in the Multi-modal interaction and transmission between 3D
objects and multiple 2D views at the same time. In order to more effectively
simultaneously perform the consistent cross-modal objective of 2D multi-view
information based on contrastive learning, we further propose Multi-MLP and
Multi-level Augmentation strategies. Through carefully designed transformation
strategies, we further learn Multi-level invariance in 2D Multi-views. MM-Point
demonstrates state-of-the-art (SOTA) performance in various downstream tasks.
For instance, it achieves a peak accuracy of 92.4% on the synthetic dataset
ModelNet40, and a top accuracy of 87.8% on the real-world dataset ScanObjectNN,
comparable to fully supervised methods. Additionally, we demonstrate its
effectiveness in tasks such as few-shot classification, 3D part segmentation
and 3D semantic segmentation.

提出一种新颖的自监督点云表示学习方法 MM-Point，通过多模态交互和传输同时处理三维物体和多个二维视图之间的一致跨模态目标，以及通过多个 MLP 和多层次增强策略来更有效地实现二维多视图信息的一致性对比学习，进一步学习了二维多视图的多层次不变性。MM-Point 在各种下游任务中展现出最先进的性能，如在合成数据集 ModelNet40 上达到了 92.4% 的峰值准确率，在真实数据集 ScanObjectNN 上达到了 87.8% 的最高准确率，与全监督方法可媲美。此外，我们还展示了它在少样本分类、三维部分分割和三维语义分割等任务中的有效性。

MM-Point：多视图信息增强的多模态自监督 3D 点云理解

MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D  Point Cloud Understanding

This paper tries to address a fundamental question in point cloud
self-supervised learning: what is a good signal we should leverage to learn
features from point clouds without annotations? To answer that, we introduce a
point cloud representation learning framework, based on geometric feature
reconstruction. In contrast to recent papers that directly adopt masked
autoencoder (MAE) and only predict original coordinates or occupancy from
masked point clouds, our method revisits differences between images and point
clouds and identifies three self-supervised learning objectives peculiar to
point clouds, namely centroid prediction, normal estimation, and curvature
prediction. Combined with occupancy prediction, these four objectives yield an
nontrivial self-supervised learning task and mutually facilitate models to
better reason fine-grained geometry of point clouds. Our pipeline is
conceptually simple and it consists of two major steps: first, it randomly
masks out groups of points, followed by a Transformer-based point cloud
encoder; second, a lightweight Transformer decoder predicts centroid, normal,
and curvature for points in each voxel. We transfer the pre-trained Transformer
encoder to a downstream peception model. On the nuScene Datset, our model
achieves 3.38 mAP improvment for object detection, 2.1 mIoU gain for
segmentation, and 1.7 AMOTA gain for multi-object tracking. We also conduct
experiments on the Waymo Open Dataset and achieve significant performance
improvements over baselines as well.

本文介绍了一种基于几何特征重建的点云表示学习框架，其中包括三个点云专有的自监督学习目标，即质心预测、表面法线估计和曲率预测。该框架采用自监督学习方法，创造有意义的任务，相互促进模型更好地推理点云的精细几何结构。通过训练后，该模型在物体检测、分割和多目标跟踪等任务上均实现了显著的表现提升。