With the rapid advancement of large language models (LLMs) and vision-language models (VLMs), significant progress has been made in developing open-vocabulary robotic manipulation systems. However, many existing approaches overlook the importance of object dynamics, limiting their applicability to more complex, dynamic tasks. In this work, we introduce KUDA, an open-vocabulary manipulation system that integrates dynamics learning and visual prompting through keypoints, leveraging both VLMs and learning-based neural dynamics models. Our key insight is that a keypoint-based target specification is simultaneously interpretable by VLMs and can be efficiently translated into cost functions for model-based planning. Given language instructions and visual observations, KUDA first assigns keypoints to the RGB image and queries the VLM to generate target specifications. These abstract keypoint-based representations are then converted into cost functions, which are optimized using a learned dynamics model to produce robotic trajectories. We evaluate KUDA on a range of manipulation tasks, including free-form language instructions across diverse object categories, multi-object interactions, and deformable or granular objects, demonstrating the effectiveness of our framework. The project page is available at http://kuda-dynamics.github.io.

本研究解决了当前开放词汇机器人操控系统在动态任务中忽视物体动态的局限性。我们提出的KUDA系统通过关键点将动态学习与视觉提示相结合，提供了一种新的目标规范方法，能够有效地将其转化为模型规划中的成本函数。研究结果表明，KUDA在多种操控任务中表现出色，展示了其在开放词汇环境下的广泛应用潜力。

KUDA：关键点统一动态学习与视觉提示的开放词汇机器人操控