Remarkable progress has been made in recent years in the fields of vision,
language, and robotics. We now have vision models capable of recognizing
objects based on language queries, navigation systems that can effectively
control mobile systems, and grasping models that can handle a wide range of
objects. Despite these advancements, general-purpose applications of robotics
still lag behind, even though they rely on these fundamental capabilities of
recognition, navigation, and grasping. In this paper, we adopt a systems-first
approach to develop a new Open Knowledge-based robotics framework called
OK-Robot. By combining Vision-Language Models (VLMs) for object detection,
navigation primitives for movement, and grasping primitives for object
manipulation, OK-Robot offers a integrated solution for pick-and-drop
operations without requiring any training. To evaluate its performance, we run
OK-Robot in 10 real-world home environments. The results demonstrate that
OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks,
representing a new state-of-the-art in Open Vocabulary Mobile Manipulation
(OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered
environments, OK-Robot's performance increases to 82%. However, the most
important insight gained from OK-Robot is the critical role of nuanced details
when combining Open Knowledge systems like VLMs with robotic modules. Videos of
our experiments are available on our website: this https URL

通过将视觉 - 语言模型（VLMs）与导航原语和抓取原语相结合，OK-Robot 提供了一个集成的解决方案，用于无需任何训练的拾取和放置操作，其在 10 个真实家居环境中取得了 58.5% 的成功率，在清洁、无杂乱环境中提高到 82%，这代表了开放词汇移动操纵中新的最先进技术。