3D scene understanding has gained significant attention due to its wide range
of applications. However, existing methods for 3D scene understanding are
limited to specific downstream tasks, which hinders their practicality in
real-world applications. This paper presents Chat-3D, which combines the 3D
visual perceptual ability of pre-trained 3D representations and the impressive
reasoning and conversation capabilities of advanced LLMs to achieve the first
universal dialogue systems for 3D scenes. Specifically, we align 3D
representations into the feature space of LLMs, thus enabling LLMs to perceive
the 3D world. Given the scarcity of 3D scene-text data, we propose a
three-stage training strategy to efficiently utilize the available data for
better alignment. To enhance the reasoning ability and develop a user-friendly
interaction scheme, we further construct a high-quality object-centric 3D
instruction dataset and design an associated object-centric prompt. Our
experiments show that Chat-3D achieves an impressive ability to comprehend
diverse instructions for 3D scenes, engage in intricate spatial reasoning, and
incorporate external knowledge into its responses. Chat-3D achieves a 75.6%
relative score compared with GPT-4 on the constructed instruction dataset.

本文介绍了 Chat-3D，它通过将预训练的 3D 表示与先进的 LLM 的推理和对话能力相结合，实现了首个用于 3D 场景的通用对话系统。 Chat-3D 能够感知 3D 世界，具备理解各种 3D 场景指令、进行复杂的空间推理并将外部知识融入其响应的能力。在构建的指令数据集上，Chat-3D 相对于 GPT-4 获得了相对得分 75.6%。