If robots are to work effectively alongside people, they must be able to
interpret natural language references to objects in their 3D environment.
Understanding 3D referring expressions is challenging -- it requires the
ability to both parse the 3D structure of the scene and correctly ground
free-form language in the presence of distraction and clutter. We introduce
Transcrib3D, an approach that brings together 3D detection methods and the
emergent reasoning capabilities of large language models (LLMs). Transcrib3D
uses text as the unifying medium, which allows us to sidestep the need to learn
shared representations connecting multi-modal inputs, which would require
massive amounts of annotated 3D data. As a demonstration of its effectiveness,
Transcrib3D achieves state-of-the-art results on 3D reference resolution
benchmarks, with a great leap in performance from previous multi-modality
baselines. To improve upon zero-shot performance and facilitate local
deployment on edge computers and robots, we propose self-correction for
fine-tuning that trains smaller models, resulting in performance close to that
of large models. We show that our method enables a real robot to perform
pick-and-place tasks given queries that contain challenging referring
expressions. Project site is at this https URL

如果机器人要与人类有效地共同工作，它们必须能够解释其 3D 环境中的物体的自然语言指示。Transcrib3D 是一种结合了 3D 检测方法和大型语言模型（LLMs）新兴推理能力的方法，通过使用文本作为统一介质，绕过了学习连接多模态输入的共享表示所需的大量注释 3D 数据的需求。通过在 3D 参考解析基准上取得最先进的结果，Transcrib3D 的有效性得到了证明，其性能远远超过了以前的多模态基准水平。为了改进零 - shot 性能并在边缘计算机和机器人上便于本地部署，我们提出了自我校正方法进行微调，训练更小的模型，结果使得性能接近于大模型的性能。我们展示了我们的方法使得真实的机器人能够在包含具有挑战性的指代表达的查询下进行拾取和放置任务。项目网站在此 https URL。