Autonomous robot navigation and manipulation in open environments require
reasoning and replanning with closed-loop feedback. We present COME-robot, the
first closed-loop framework utilizing the GPT-4V vision-language foundation
model for open-ended reasoning and adaptive planning in real-world scenarios.
We meticulously construct a library of action primitives for robot exploration,
navigation, and manipulation, serving as callable execution modules for GPT-4V
in task planning. On top of these modules, GPT-4V serves as the brain that can
accomplish multimodal reasoning, generate action policy with code, verify the
task progress, and provide feedback for replanning. Such design enables
COME-robot to (i) actively perceive the environments, (ii) perform situated
reasoning, and (iii) recover from failures. Through comprehensive experiments
involving 8 challenging real-world tabletop and manipulation tasks, COME-robot
demonstrates a significant improvement in task success rate (~25%) compared to
state-of-the-art baseline methods. We further conduct comprehensive analyses to
elucidate how COME-robot's design facilitates failure recovery, free-form
instruction following, and long-horizon task planning.

使用 GPT-4V 视觉语言基础模型的 CLOSE-robot 闭环框架，在开放环境中进行自主机器人导航和操作，通过多模态推理、生成动作策略、验证任务进展和提供反馈，实现了环境感知、情境推理和故障恢复，显著提升了任务成功率和故障恢复能力。