We describe an approach to predict open-vocabulary 3D semantic voxel
occupancy map from input 2D images with the objective of enabling 3D grounding,
segmentation and retrieval of free-form language queries. This is a challenging
problem because of the 2D-3D ambiguity and the open-vocabulary nature of the
target tasks, where obtaining annotated training data in 3D is difficult. The
contributions of this work are three-fold. First, we design a new model
architecture for open-vocabulary 3D semantic occupancy prediction. The
architecture consists of a 2D-3D encoder together with occupancy prediction and
3D-language heads. The output is a dense voxel map of 3D grounded language
embeddings enabling a range of open-vocabulary tasks. Second, we develop a
tri-modal self-supervised learning algorithm that leverages three modalities:
(i) images, (ii) language and (iii) LiDAR point clouds, and enables training
the proposed architecture using a strong pre-trained vision-language model
without the need for any 3D manual language annotations. Finally, we
demonstrate quantitatively the strengths of the proposed model on several
open-vocabulary tasks: Zero-shot 3D semantic segmentation using existing
datasets; 3D grounding and retrieval of free-form language queries, using a
small dataset that we propose as an extension of nuScenes. You can find the
project page here this https URL

我们描述了一种预测开放词汇的 3D 语义体素占用图的方法，该方法可实现对自由形式语言查询进行 3D 定位、分割和检索。我们设计了一种新的模型架构，包括 2D-3D 编码器、占用预测和 3D 语言头；开发了一种三模态自监督学习算法，利用图像、语言和 LiDAR 点云三种模态来训练所提出的架构；并在几个开放词汇任务上定量展示了所提出模型的优势。

POP-3D：基于图像的开放词汇 3D 实占预测

POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images

The remarkable potential of multi-modal large language models (MLLMs) in
comprehending both vision and language information has been widely
acknowledged. However, the scarcity of 3D scenes-language pairs in comparison
to their 2D counterparts, coupled with the inadequacy of existing approaches in
understanding of 3D scenes by LLMs, poses a significant challenge. In response,
we collect and construct an extensive dataset comprising 75K
instruction-response pairs tailored for 3D scenes. This dataset addresses tasks
related to 3D VQA, 3D grounding, and 3D conversation. To further enhance the
integration of 3D spatial information into LLMs, we introduce a novel and
efficient prompt tuning paradigm, 3DMIT. This paradigm eliminates the alignment
stage between 3D scenes and language and extends the instruction prompt with
the 3D modality information including the entire scene and segmented objects.
We evaluate the effectiveness of our method across diverse tasks in the 3D
scene domain and find that our approach serves as a strategic means to enrich
LLMs' comprehension of the 3D world. Our code is available at
this https URL

我们构建了一个包含 75K 个指令 - 回答对的广泛数据集，针对 3D 场景进行了任务，如 3D VQA、3D grounding 和 3D conversation，并引入了一种名为 3DMIT 的新颖且高效的提示调优范例，通过将 3D 空间信息整合到 LLMs 中，使其更好地理解 3D 世界。

3DMIT: 3D 多模态指导调整用于场景理解

3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding

Grounding object properties and relations in 3D scenes is a prerequisite for
a wide range of artificial intelligence tasks, such as visually grounded
dialogues and embodied manipulation. However, the variability of the 3D domain
induces two fundamental challenges: 1) the expense of labeling and 2) the
complexity of 3D grounded language. Hence, essential desiderata for models are
to be data-efficient, generalize to different data distributions and tasks with
unseen semantic forms, as well as ground complex language semantics (e.g.,
view-point anchoring and multi-object reference). To address these challenges,
we propose NS3D, a neuro-symbolic framework for 3D grounding. NS3D translates
language into programs with hierarchical structures by leveraging large
language-to-code models. Different functional modules in the programs are
implemented as neural networks. Notably, NS3D extends prior neuro-symbolic
visual reasoning methods by introducing functional modules that effectively
reason about high-arity relations (i.e., relations among more than two
objects), key in disambiguating objects in complex 3D scenes. Modular and
compositional architecture enables NS3D to achieve state-of-the-art results on
the ReferIt3D view-dependence task, a 3D referring expression comprehension
benchmark. Importantly, NS3D shows significantly improved performance on
settings of data-efficiency and generalization, and demonstrate zero-shot
transfer to an unseen 3D question-answering task.

本文提出了 NS3D 三维场景认知神经符号框架，通过使用基于大规模语言编程模型的层级结构来将语言翻译为程序，不同的功能模块实现为神经网络，并引入管高度关系的功能模块有效地推理复杂场景中物体之间的关系，在数据效率和泛化方面表现出色，在 3D 视角相关任务 ReferIt3D 上获得了最先进的结果。