Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for capturing essential object features, and for establishing a reference frame in action prediction, enabling data-efficient learning of robot skills. However, their manual design nature and reliance on additional human labels limit their scalability. In this paper, we propose KALM, a framework that leverages large pre-trained vision-language models (LMs) to automatically generate task-relevant and cross-instance consistent keypoints. KALM distills robust and consistent keypoints across views and objects by generating proposals using LMs and verifies them against a small set of robot demonstration data. Based on the generated keypoints, we can train keypoint-conditioned policy models that predict actions in keypoint-centric frames, enabling robots to generalize effectively across varying object poses, camera views, and object instances with similar functional shapes. Our method demonstrates strong performance in the real world, adapting to different tasks and environments from only a handful of demonstrations while requiring no additional labels. Website: https://kalm-il.github.io/

本研究解决了机器人在多样化任务和环境中对新物体配置和实例的泛化挑战。提出的KALM框架利用大型预训练的视觉-语言模型自动生成任务相关且跨实例一致的关键点，从而提高机器人技能学习的效率。实验表明，该方法在真实环境中表现强劲，能够在少量演示的情况下适应不同的任务和环境，无需额外标签。

使用大型模型进行物体相关模仿学习的关键点抽象