The ability to grasp objects in-the-wild from open-ended language
instructions constitutes a fundamental challenge in robotics. An open-world
grasping system should be able to combine high-level contextual with low-level
physical-geometric reasoning in order to be applicable in arbitrary scenarios.
Recent works exploit the web-scale knowledge inherent in large language models
(LLMs) to plan and reason in robotic context, but rely on external vision and
action models to ground such knowledge into the environment and parameterize
actuation. This setup suffers from two major bottlenecks: a) the LLM's
reasoning capacity is constrained by the quality of visual grounding, and b)
LLMs do not contain low-level spatial understanding of the world, which is
essential for grasping in contact-rich scenarios. In this work we demonstrate
that modern vision-language models (VLMs) are capable of tackling such
limitations, as they are implicitly grounded and can jointly reason about
semantics and geometry. We propose OWG, an open-world grasping pipeline that
combines VLMs with segmentation and grasp synthesis models to unlock grounded
world understanding in three stages: open-ended referring segmentation,
grounded grasp planning and grasp ranking via contact reasoning, all of which
can be applied zero-shot via suitable visual prompting mechanisms. We conduct
extensive evaluation in cluttered indoor scene datasets to showcase OWG's
robustness in grounding from open-ended language, as well as open-world robotic
grasping experiments in both simulation and hardware that demonstrate superior
performance compared to previous supervised and zero-shot LLM-based methods.

通过结合语言模型 (vision-language models)、分割模型和抓取综合模型，提出了一个能够在任何场景中理解、规划和执行物体抓取的开放世界抓取系统 (OWG)，通过视觉提示机制，借助于开放式自由语言的引导实现零样本应用，展示了该系统在杂乱室内场景数据集和仿真环境、硬件实验中相对于以往的有监督和零样本语言模型方法具有优越性能。