Recent success of vision foundation models have shown promising performance
for the 2D perception tasks. However, it is difficult to train a 3D foundation
network directly due to the limited dataset and it remains under explored
whether existing foundation models can be lifted to 3D space seamlessly. In
this paper, we present PointSeg, a novel training-free paradigm that leverages
off-the-shelf vision foundation models to address 3D scene perception tasks.
PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to
align their corresponding pixels across frames. Concretely, we design a
two-branch prompts learning structure to construct the 3D point-box prompts
pairs, combining with the bidirectional matching strategy for accurate point
and proposal prompts generation. Then, we perform the iterative post-refinement
adaptively when cooperated with different vision foundation models. Moreover,
we design a affinity-aware merging algorithm to improve the final ensemble
masks. PointSeg demonstrates impressive segmentation performance across various
datasets, all without training. Specifically, our approach significantly
surpasses the state-of-the-art specialist model by 13.4$\%$, 11.3$\%$, and
12$\%$ mAP on ScanNet, ScanNet++, and KITTI-360 datasets, respectively. On top
of that, PointSeg can incorporate with various segmentation models and even
surpasses the supervised methods.

PointSeg 是一种新的无需训练的范例，借助现有的视觉基础模型来解决 3D 场景感知任务，并展示了令人印象深刻的分割性能。

PointSeg：基于基础模型的三维场景分割的非训练范式

PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via  Foundation Models

In this paper, we propose PointSeg, a real-time end-to-end semantic
segmentation method for road-objects based on spherical images. We take the
spherical image, which is transformed from the 3D LiDAR point clouds, as input
of the convolutional neural networks (CNNs) to predict the point-wise semantic
map. To make PointSeg applicable on a mobile system, we build the model based
on the light-weight network, SqueezeNet, with several improvements. It
maintains a good balance between memory cost and prediction performance. Our
model is trained on spherical images and label masks projected from the KITTI
3D object detection dataset. Experiments show that PointSeg can achieve
competitive accuracy with 90fps on a single GPU 1080ti. which makes it quite
compatible for autonomous driving applications.

本文提出了一种基于球形图像的实时端到端语义分割方法 PointSeg，它以从 3D LiDAR 点云转换来的球形图像为输入，并利用卷积神经网络（CNNs）来预测点级语义映射，它采用了轻量级网络 SqueezeNet 进行模型构建，并在 KITTI 3D 目标检测数据集上进行了训练，实验结果表明在单个 GPU 1080ti 上可达到每秒 90 帧的竞争精度，适用于自动驾驶应用。