Open-vocabulary 3D scene understanding presents a significant challenge in
the field. Recent advancements have sought to transfer knowledge embedded in
vision language models from the 2D domain to 3D domain. However, these
approaches often require learning prior knowledge from specific 3D scene
datasets, which limits their applicability in open-world scenarios. The Segment
Anything Model (SAM) has demonstrated remarkable zero-shot segmentation
capabilities, prompting us to investigate its potential for comprehending 3D
scenes without the need for training. In this paper, we introduce OV-SAM3D, a
universal framework for open-vocabulary 3D scene understanding. This framework
is designed to perform understanding tasks for any 3D scene without requiring
prior knowledge of the scene. Specifically, our method is composed of two key
sub-modules: First, we initiate the process by generating superpoints as the
initial 3D prompts and refine these prompts using segment masks derived from
SAM. Moreover, we then integrate a specially designed overlapping score table
with open tags from the Recognize Anything Model (RAM) to produce final 3D
instances with open-world label. Empirical evaluations conducted on the
ScanNet200 and nuScenes datasets demonstrate that our approach surpasses
existing open-vocabulary methods in unknown open-world environments.

OV-SAM3D 是一个通用框架，用于不需要训练即可理解任何 3D 场景的开放词汇三维场景理解，通过使用 Segment Anything Model (SAM) 生成超点并通过 Recognize Anything Model (RAM) 的开放标签和操作表，结合超点和分割掩模生成最终的 3D 实例，经过对 ScanNet200 和 nuScenes 数据集的实证评估，我们的方法在未知的开放世界环境中超越了现有的开放词汇方法。

开放词汇 SAM3D：理解任何三维场景

Open-Vocabulary SAM3D: Understand Any 3D Scene

We study the problem of building a controller that can follow open-ended
instructions in open-world environments. We propose to follow reference videos
as instructions, which offer expressive goal specifications while eliminating
the need for expensive text-gameplay annotations. A new learning framework is
derived to allow learning such instruction-following controllers from gameplay
videos while producing a video instruction encoder that induces a structured
goal space. We implement our agent GROOT in a simple yet effective
encoder-decoder architecture based on causal transformers. We evaluate GROOT
against open-world counterparts and human players on a proposed Minecraft
SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the
human-machine gap as well as exhibiting a 70% winning rate over the best
generalist agent baseline. Qualitative analysis of the induced goal space
further demonstrates some interesting emergent properties, including the goal
composition and complex gameplay behavior synthesis. Code and video can be
found on the website this https URL

通过引入视频作为指导说明，我们研究了在开放世界环境中构建能够跟随无限指令的控制器的问题，并提出了一种全新的学习框架，该框架能够从游戏视频中学习这种跟随指令的控制器，并生成一个诱导结构化目标空间的视频指令编码器。通过对提出的 Minecraft SkillForge 基准上的开放世界对手和人类玩家的评估，我们将我们的代理 GROOT 与其他对手进行了比较，并且评分清晰地表明 GROOT 正在缩小人机差距，并且在最佳综合代理基准上展现出 70% 的胜率。对诱发的目标空间进行的定性分析进一步展示了一些有趣的新特性，包括目标组合和复杂的游戏行为合成。