Large Language Models (LLMs) have gained popularity in task planning for long-horizon manipulation tasks. To enhance the validity of LLM-generated plans, visual demonstrations and online videos have been widely employed to guide the planning process. However, for manipulation tasks involving subtle movements but rich contact interactions, visual perception alone may be insufficient for the LLM to fully interpret the demonstration. Additionally, visual data provides limited information on force-related parameters and conditions, which are crucial for effective execution on real robots. In this paper, we introduce an in-context learning framework that incorporates tactile and force-torque information from human demonstrations to enhance LLMs' ability to generate plans for new task scenarios. We propose a bootstrapped reasoning pipeline that sequentially integrates each modality into a comprehensive task plan. This task plan is then used as a reference for planning in new task configurations. Real-world experiments on two different sequential manipulation tasks demonstrate the effectiveness of our framework in improving LLMs' understanding of multi-modal demonstrations and enhancing the overall planning performance.

本研究解决了在复杂接触操作任务中，单靠视觉信息不足以充分理解演示的问题。通过引入触觉和力-扭矩信息，提出了一种新的在上下文学习框架，增强了大型语言模型生成新任务情景的规划能力。实验表明，该框架显著提升了多模态演示的理解和规划性能，为实际机器人操作提供了更有效的支持。

从多模态演示学习任务规划以实现多阶段接触丰富的操作