From rearranging objects on a table to putting groceries into shelves, robots
must plan precise action points to perform tasks accurately and reliably. In
spite of the recent adoption of vision language models (VLMs) to control robot
behavior, VLMs struggle to precisely articulate robot actions using language.
We introduce an automatic synthetic data generation pipeline that
instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we
train RoboPoint, a VLM that predicts image keypoint affordances given language
instructions. Compared to alternative approaches, our method requires no
real-world data collection or human demonstration, making it much more scalable
to diverse environments and viewpoints. In addition, RoboPoint is a general
model that enables several downstream applications such as robot navigation,
manipulation, and augmented reality (AR) assistance. Our experiments
demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and
visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting
spatial affordance and by 30.5% in the success rate of downstream tasks.
Project website: this https URL

我们介绍了一个自动的合成数据生成流水线，用于调整机器人领域和需求中的视觉语言模型，并通过该流水线训练了 RoboPoint，该模型可以根据语言指令预测图像关键点的可行性。与其他方法相比，我们的方法无需真实世界数据收集或人类示范，使得它在适应多样环境和视点方面具有更大的可扩展性。此外，RoboPoint 是一个通用模型，可以用于机器人导航、操作和增强现实（AR）辅助等多个下游应用。我们的实验证明，RoboPoint 在预测空间可行性的准确性上比最先进的 VLMs（GPT-4o）和视觉提示技术（PIVOT）提高了 21.8％，在下游任务的成功率上提高了 30.5％。