We propose a novel approach to multi-robot collaboration that harnesses the
power of pre-trained large language models (LLMs) for both high-level
communication and low-level path planning. Robots are equipped with LLMs to
discuss and collectively reason task strategies. They then generate sub-task
plans and task space waypoint paths, which are used by a multi-arm motion
planner to accelerate trajectory planning. We also provide feedback from the
environment, such as collision checking, and prompt the LLM agents to improve
their plan and waypoints in-context. For evaluation, we introduce RoCoBench, a
6-task benchmark covering a wide range of multi-robot collaboration scenarios,
accompanied by a text-only dataset for agent representation and reasoning. We
experimentally demonstrate the effectiveness of our approach -- it achieves
high success rates across all tasks in RoCoBench and adapts to variations in
task semantics. Our dialog setup offers high interpretability and flexibility
-- in real world experiments, we show RoCo easily incorporates
human-in-the-loop, where a user can communicate and collaborate with a robot
agent to complete tasks together. See project website
this https URL for videos and code.

我们提出了一种新颖的多机器人协作方法，利用预训练的大型语言模型（LLMs）进行高层通信和低层路径规划，通过机器人之间的交流和集体推理任务策略，并生成子任务计划和任务空间路径，应用于多臂运动规划，以加速轨迹规划，并在环境中提供反馈，如碰撞检测，以促使 LLM 代理改进其计划和路径点。我们提出了 RoCoBench，一个包含六个任务的广泛多机器人协作场景的基准测试，伴随着一个纯文本数据集用于代理表示和推理。我们在实验证明了我们方法的有效性 - 它在 RoCoBench 的所有任务上都取得了高成功率，并且能够适应任务语义的变化。我们的对话设置提供了高度的可解释性和灵活性 - 在真实世界的实验中，我们展示了 RoCo 可以轻松地与人工智能交互，用户可以与机器人代理合作完成任务。请参考项目网站以观看视频和获取代码。