3D Visual Grounding (3DVG) aims at localizing 3D object based on textual
descriptions. Conventional supervised methods for 3DVG often necessitate
extensive annotations and a predefined vocabulary, which can be restrictive. To
address this issue, we propose a novel visual programming approach for
zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language
models (LLMs). Our approach begins with a unique dialog-based method, engaging
with LLMs to establish a foundational understanding of zero-shot 3DVG. Building
on this, we design a visual program that consists of three types of modules,
i.e., view-independent, view-dependent, and functional modules. These modules,
specifically tailored for 3D scenarios, work collaboratively to perform complex
reasoning and inference. Furthermore, we develop an innovative language-object
correlation module to extend the scope of existing 3D object detectors into
open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot
approach can outperform some supervised baselines, marking a significant stride
towards effective 3DVG.

通过使用大型语言模型，我们提出了一种零样本开放词汇的三维视觉定位方法，使用独特的基于对话的方法和视觉程序模块，以及创新的语言 - 物体关联模块，在三维场景中实现了复杂推理，并扩展了现有三维物体检测器的应用范围，取得了显著的性能优势。