While large language models (LLMs) are successful in completing various language processing tasks, they easily fail to interact with the physical world by generating control sequences properly. We find that the main reason is that LLMs are not grounded in the physical world. Existing LLM-based approaches circumvent this problem by relying on additional pre-defined skills or pre-trained sub-policies, making it hard to adapt to new tasks. In contrast, we aim to address this problem and explore the possibility to prompt pre-trained LLMs to accomplish a series of robotic manipulation tasks in a training-free paradigm. Accordingly, we propose a framework called LLM+A(ffordance) where the LLM serves as both the sub-task planner (that generates high-level plans) and the motion controller (that generates low-level control sequences). To ground these plans and control sequences on the physical world, we develop the affordance prompting technique that stimulates the LLM to 1) predict the consequences of generated plans and 2) generate affordance values for relevant objects. Empirically, we evaluate the effectiveness of LLM+A in various language-conditioned robotic manipulation tasks, which show that our approach substantially improves performance by enhancing the feasibility of generated plans and control and can easily generalize to different environments.

大型语言模型在完成各种语言处理任务方面很成功，但在生成适当的控制序列时很容易与物理世界互动失败。我们发现主要原因是LLM没有与物理世界紧密连接。与现有基于LLM的方法相比，我们旨在解决这个问题，并探索在无训练范式下促使预训练LLM完成一系列机器人操作任务的可能性。为了将这些计划和控制序列与物理世界联系起来，我们开发了促使技术，通过刺激LLM来预测生成计划的后果并为相关对象生成可供性值。实验证明，LLM+A在各种语言条件下的机器人操作任务中的有效性，显示了我们的方法通过增强生成计划和控制的可行性来显著提高性能，并能够轻松推广到不同的环境。

利用能力促进大型语言模型在机器人操纵方面的应用