Recent success of vision foundation models have shown promising performance
for the 2D perception tasks. However, it is difficult to train a 3D foundation
network directly due to the limited dataset and it remains under explored
whether existing foundation models can be lifted to 3D space seamlessly. In
this paper, we present PointSeg, a novel training-free paradigm that leverages
off-the-shelf vision foundation models to address 3D scene perception tasks.
PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to
align their corresponding pixels across frames. Concretely, we design a
two-branch prompts learning structure to construct the 3D point-box prompts
pairs, combining with the bidirectional matching strategy for accurate point
and proposal prompts generation. Then, we perform the iterative post-refinement
adaptively when cooperated with different vision foundation models. Moreover,
we design a affinity-aware merging algorithm to improve the final ensemble
masks. PointSeg demonstrates impressive segmentation performance across various
datasets, all without training. Specifically, our approach significantly
surpasses the state-of-the-art specialist model by 13.4$\%$, 11.3$\%$, and
12$\%$ mAP on ScanNet, ScanNet++, and KITTI-360 datasets, respectively. On top
of that, PointSeg can incorporate with various segmentation models and even
surpasses the supervised methods.

PointSeg 是一种新的无需训练的范例，借助现有的视觉基础模型来解决 3D 场景感知任务，并展示了令人印象深刻的分割性能。

PointSeg：基于基础模型的三维场景分割的非训练范式

PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via  Foundation Models

Current successful methods of 3D scene perception rely on the large-scale
annotated point cloud, which is tedious and expensive to acquire. In this
paper, we propose Model2Scene, a novel paradigm that learns free 3D scene
representation from Computer-Aided Design (CAD) models and languages. The main
challenges are the domain gaps between the CAD models and the real scene's
objects, including model-to-scene (from a single model to the scene) and
synthetic-to-real (from synthetic model to real scene's object). To handle the
above challenges, Model2Scene first simulates a crowded scene by mixing
data-augmented CAD models. Next, we propose a novel feature regularization
operation, termed Deep Convex-hull Regularization (DCR), to project point
features into a unified convex hull space, reducing the domain gap. Ultimately,
we impose contrastive loss on language embedding and the point features of CAD
models to pre-train the 3D network. Extensive experiments verify the learned 3D
scene representation is beneficial for various downstream tasks, including
label-free 3D object salient detection, label-efficient 3D scene perception and
zero-shot 3D semantic segmentation. Notably, Model2Scene yields impressive
label-free 3D object salient detection with an average mAP of 46.08\% and
55.49\% on the ScanNet and S3DIS datasets, respectively. The code will be
publicly available.

当前成功的三维场景感知方法依赖于大规模标注的点云，获取费时费力。本文提出了 Model2Scene，一种新颖的范式，从计算机辅助设计 (CAD) 模型和语言中学习自由的三维场景表示。主要挑战是 CAD 模型与真实场景对象之间的领域差异，包括模型到场景的转换和合成模型到真实场景对象的转换。为了应对上述挑战，Model2Scene 首先通过混合数据增强的 CAD 模型模拟拥挤的场景。然后，我们提出了一种称为深凸包正则化 (DCR) 的新颖特征正则化操作，将点特征投影到统一的凸包空间，减小领域差距。最后，我们对 CAD 模型的语言嵌入和点特征进行对比损失的预训练，形成 3D 网络。大量实验证实了学习到的三维场景表示对各种下游任务的益处，包括无标签三维物体显著目标检测、标签高效三维场景感知和零样本三维语义分割。值得注意的是，Model2Scene 在 ScanNet 和 S3DIS 数据集上实现了令人印象深刻的无标签三维物体显著目标检测，平均 mAP 分别为 46.08% 和 55.49%。代码将公开提供。

Model2Scene: 通过对比学习语言 - CAD 模型预训练实现 3D 场景表示

Model2Scene: Learning 3D Scene Representation via Contrastive  Language-CAD Models Pre-training

Project AutoVision aims to develop localization and 3D scene perception
capabilities for a self-driving vehicle. Such capabilities will enable
autonomous navigation in urban and rural environments, in day and night, and
with cameras as the only exteroceptive sensors. The sensor suite employs many
cameras for both 360-degree coverage and accurate multi-view stereo; the use of
low-cost cameras keeps the cost of this sensor suite to a minimum. In addition,
the project seeks to extend the operating envelope to include GNSS-less
conditions which are typical for environments with tall buildings, foliage, and
tunnels. Emphasis is placed on leveraging multi-view geometry and deep learning
to enable the vehicle to localize and perceive in 3D space. This paper presents
an overview of the project, and describes the sensor suite and current progress
in the areas of calibration, localization, and perception.

Project AutoVision 旨在为自动驾驶汽车开发定位和 3D 场景感知的能力，利用低成本多相机的传感器套件实现 360 度覆盖和准确的多视图立体感知，在使用深度学习和多视角几何的帮助下，提高车辆在城市和乡村地区等各种环境下的自主导航。