3D vision-language grounding, which focuses on aligning language with the 3D
physical environment, stands as a cornerstone in the development of embodied
agents. In comparison to recent advancements in the 2D domain, grounding
language in 3D scenes faces several significant challenges: (i) the inherent
complexity of 3D scenes due to the diverse object configurations, their rich
attributes, and intricate relationships; (ii) the scarcity of paired 3D
vision-language data to support grounded learning; and (iii) the absence of a
unified learning framework to distill knowledge from grounded 3D data. In this
work, we aim to address these three major challenges in 3D vision-language by
examining the potential of systematically upscaling 3D vision-language learning
in indoor environments. We introduce the first million-scale 3D vision-language
dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising
2.5M vision-language pairs derived from both human annotations and our scalable
scene-graph-based generation approach. We demonstrate that this scaling allows
for a unified pre-training framework, Grounded Pre-training for Scenes (GPS),
for 3D vision-language learning. Through extensive experiments, we showcase the
effectiveness of GPS by achieving state-of-the-art performance on all existing
3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is
unveiled through zero-shot transfer experiments in the challenging 3D
vision-language tasks. Project website: this https URL .

通过系统性地将 3D 视觉语言学习在室内环境中进行有序提升，本研究旨在解决 3D 视觉语言面临的三个主要挑战，包括复杂的 3D 场景、缺乏数据支持和缺乏统一的学习框架，并通过引入包含约 68K 个 3D 室内场景的场景语料库 SceneVerse 以及基于可扩展的场景图生成方法获取的约 2.5M 个视觉语言对，展示了 Grounded Pre-training for Scenes (GPS) 的有效性，通过在所有现有的 3D 视觉定位基准上取得了最先进的性能，并在具有挑战性的 3D 视觉语言任务的零样本迁移实验中揭示了 SceneVerse 和 GPS 的巨大潜力。

SceneVerse：面向基于场景的三维视觉语言学习的规模化

SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene  Understanding

3D vision-language grounding (3D-VL) is an emerging field that aims to
connect the 3D physical world with natural language, which is crucial for
achieving embodied intelligence. Current 3D-VL models rely heavily on
sophisticated modules, auxiliary losses, and optimization tricks, which calls
for a simple and unified model. In this paper, we propose 3D-VisTA, a
pre-trained Transformer for 3D Vision and Text Alignment that can be easily
adapted to various downstream tasks. 3D-VisTA simply utilizes self-attention
layers for both single-modal modeling and multi-modal fusion without any
sophisticated task-specific design. To further enhance its performance on 3D-VL
tasks, we construct ScanScribe, the first large-scale 3D scene-text pairs
dataset for 3D-VL pre-training. ScanScribe contains 2,995 RGB-D scans for 1,185
unique indoor scenes originating from ScanNet and 3R-Scan datasets, along with
paired 278K scene descriptions generated from existing 3D-VL tasks, templates,
and GPT-3. 3D-VisTA is pre-trained on ScanScribe via masked language/object
modeling and scene-text matching. It achieves state-of-the-art results on
various 3D-VL tasks, ranging from visual grounding and dense captioning to
question answering and situated reasoning. Moreover, 3D-VisTA demonstrates
superior data efficiency, obtaining strong performance even with limited
annotations during downstream task fine-tuning.

提出了 3D-VisTA，一个用于 3D 视觉和文本对齐的预训练 Transformer 模型，可轻松适应各种下游任务，通过在 ScanScribe 数据集上进行预训练，获得了在各种 3D-VL 任务上的最先进结果，同时展示了出色的数据效率。