Open-vocabulary 3D scene understanding presents a significant challenge in
the field. Recent advancements have sought to transfer knowledge embedded in
vision language models from the 2D domain to 3D domain. However, these
approaches often require learning prior knowledge from specific 3D scene
datasets, which limits their applicability in open-world scenarios. The Segment
Anything Model (SAM) has demonstrated remarkable zero-shot segmentation
capabilities, prompting us to investigate its potential for comprehending 3D
scenes without the need for training. In this paper, we introduce OV-SAM3D, a
universal framework for open-vocabulary 3D scene understanding. This framework
is designed to perform understanding tasks for any 3D scene without requiring
prior knowledge of the scene. Specifically, our method is composed of two key
sub-modules: First, we initiate the process by generating superpoints as the
initial 3D prompts and refine these prompts using segment masks derived from
SAM. Moreover, we then integrate a specially designed overlapping score table
with open tags from the Recognize Anything Model (RAM) to produce final 3D
instances with open-world label. Empirical evaluations conducted on the
ScanNet200 and nuScenes datasets demonstrate that our approach surpasses
existing open-vocabulary methods in unknown open-world environments.

OV-SAM3D 是一个通用框架，用于不需要训练即可理解任何 3D 场景的开放词汇三维场景理解，通过使用 Segment Anything Model (SAM) 生成超点并通过 Recognize Anything Model (RAM) 的开放标签和操作表，结合超点和分割掩模生成最终的 3D 实例，经过对 ScanNet200 和 nuScenes 数据集的实证评估，我们的方法在未知的开放世界环境中超越了现有的开放词汇方法。

开放词汇 SAM3D：理解任何三维场景

Open-Vocabulary SAM3D: Understand Any 3D Scene

Open-vocabulary 3D scene understanding presents a significant challenge in
computer vision, withwide-ranging applications in embodied agents and augmented
reality systems. Previous approaches haveadopted Neural Radiance Fields (NeRFs)
to analyze 3D scenes. In this paper, we introduce SemanticGaussians, a novel
open-vocabulary scene understanding approach based on 3D Gaussian Splatting.
Our keyidea is distilling pre-trained 2D semantics into 3D Gaussians. We design
a versatile projection approachthat maps various 2Dsemantic features from
pre-trained image encoders into a novel semantic component of 3D Gaussians,
withoutthe additional training required by NeRFs. We further build a 3D
semantic network that directly predictsthe semantic component from raw 3D
Gaussians for fast inference. We explore several applications ofSemantic
Gaussians: semantic segmentation on ScanNet-20, where our approach attains a
4.2% mIoU and 4.0%mAcc improvement over prior open-vocabulary scene
understanding counterparts; object part segmentation,sceneediting, and
spatial-temporal segmentation with better qualitative results over 2D and 3D
baselines,highlighting its versatility and effectiveness on supporting diverse
downstream tasks.

基于 3D 高斯喷斑的新型开放词汇场景理解方法，通过提取预训练的 2D 语义学习特征，将其融入 3D 高斯成分，并构建 3D 高斯语义网络用于快速推断，实现了在 ScanNet-20 上对语义分割的改进、对物体部分分割、场景编辑以及时空分割的多样性支持和优越性。

语义高斯：基于 3D 高斯点云绘制的开放词汇场景理解

Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian  Splatting

This report provides an overview of the challenge hosted at the OpenSUN3D
Workshop on Open-Vocabulary 3D Scene Understanding held in conjunction with
ICCV 2023. The goal of this workshop series is to provide a platform for
exploration and discussion of open-vocabulary 3D scene understanding tasks,
including but not limited to segmentation, detection and mapping. We provide an
overview of the challenge hosted at the workshop, present the challenge
dataset, the evaluation methodology, and brief descriptions of the winning
methods. For additional details, please see
this https URL

该报告提供了 OpenSUN3D 举办的 Open-Vocabulary 3D 场景理解挑战赛的概述，包括分割、检测和映射等任务，并介绍了挑战赛的数据集、评估方法和获奖方法的简要描述。