3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to decompose complex natural language queries into semantic constituents and employs a visual grounding tool, such as OpenScene or LERF, to identify objects in a 3D scene. The LLM then evaluates the spatial and commonsense relations among the proposed objects to make a final grounding decision. Our method does not require any labeled training data and can generalize to novel 3D scenes and arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and demonstrate state-of-the-art zero-shot grounding accuracy. Our findings indicate that LLMs significantly improve the grounding capability, especially for complex language queries, making LLM-Grounder an effective approach for 3D vision-language tasks in robotics. Videos and interactive demos can be found on the project website https://chat-with-nerf.github.io/ .

LLM-Grounder是一种零样本、开放词汇量的基于大型语言模型（LLM）的3D视觉定位流水线。通过将复杂自然语言查询拆解为语义元素，并利用可视化定位工具识别3D场景中的对象，LLM-Grounder评估所提议对象之间的空间和常识关系，从而做出最终的定位决策。该方法不需要有标签的培训数据，可应用于新型3D场景和任意文本查询，显示出最先进的零样本定位准确性。研究结果表明，LLM显著提高了定位能力，尤其对于复杂语言查询，在机器人的3D视觉语言任务中，LLM-Grounder是一种有效的方法。

LLM-Grounder：使用大型语言模型作为代理人进行开放词汇三维视觉对接