Recently, Large Language Models (LLMs) have achieved remarkable success using in-context learning (ICL) in the language domain. However, leveraging the ICL capabilities within LLMs to directly predict robot actions remains largely unexplored. In this paper, we introduce RoboPrompt, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions through ICL without training. Our approach first heuristically identifies keyframes that capture important moments from an episode. Next, we extract end-effector actions from these keyframes as well as the estimated initial object poses, and both are converted into textual descriptions. Finally, we construct a structured template to form ICL demonstrations from these textual descriptions and a task instruction. This enables an LLM to directly predict robot actions at test time. Through extensive experiments and analysis, RoboPrompt shows stronger performance over zero-shot and ICL baselines in simulated and real-world settings.

本研究针对现有大型语言模型（LLMs）在直接预测机器人动作方面的研究空白，提出了RoboPrompt框架。该框架利用上下文学习（ICL）技术，无需训练即可使文本基础的LLMs直接预测机器人动作。通过对关键帧的识别和描述，RoboPrompt在模拟和真实环境中展现了优于零-shot和ICL基线的性能。

上下文学习使机器人动作预测在大型语言模型中成为可能