Enabling embodied agents to complete complex human instructions from natural
language is crucial to autonomous systems in household services. Conventional
methods can only accomplish human instructions in the known environment where
all interactive objects are provided to the embodied agent, and directly
deploying the existing approaches for the unknown environment usually generates
infeasible plans that manipulate non-existing objects. On the contrary, we
propose an embodied instruction following (EIF) method for complex tasks in the
unknown environment, where the agent efficiently explores the unknown
environment to generate feasible plans with existing objects to accomplish
abstract instructions. Specifically, we build a hierarchical embodied
instruction following framework including the high-level task planner and the
low-level exploration controller with multimodal large language models. We then
construct a semantic representation map of the scene with dynamic region
attention to demonstrate the known visual clues, where the goal of task
planning and scene exploration is aligned for human instruction. For the task
planner, we generate the feasible step-by-step plans for human goal
accomplishment according to the task completion process and the known visual
clues. For the exploration controller, the optimal navigation or object
interaction policy is predicted based on the generated step-wise plans and the
known visual clues. The experimental results demonstrate that our method can
achieve 45.09% success rate in 204 complex human instructions such as making
breakfast and tidying rooms in large house-level scenes.

通过利用多模态大型语言模型，我们构建了一个层次化的具有高级任务规划器和低级探索控制器的嵌入指令跟随框架，该方法可以在未知环境中通过在已知可见线索的基础上生成可行的计划来完成抽象指令，实验结果表明，在大型房屋场景中，我们的方法在 204 个复杂的人类指令（如做早餐和整理房间）中实现了 45.09% 的成功率。

未知环境中的身体指令遵循

Embodied Instruction Following in Unknown Environments

The theoretical ability of modular robots to reconfigure in response to
complex tasks in a priori unknown environments has frequently been cited as an
advantage and remains a major motivator for work in the field. We present a
modular robot system capable of autonomously completing high-level tasks by
reactively reconfiguring to meet the needs of a perceived, a priori unknown
environment. The system integrates perception, high-level planning, and modular
hardware, and is validated in three hardware demonstrations. Given a high-level
task specification, a modular robot autonomously explores an unknown
environment, decides when and how to reconfigure, and manipulates objects to
complete its task. The system architecture balances distributed mechanical
elements with centralized perception, planning, and control. By providing an
example of how a modular robot system can be designed to leverage reactive
reconfigurability in unknown environments, we have begun to lay the groundwork
for modular self-reconfigurable robots to address tasks in the real world.

通过在未知环境下实现感知、高层规划和模块化硬件的集成，我们展示了一种模块化机器人系统，能够通过反应式重新配置来完成高级任务。该系统在三次硬件演示中得到验证，为模块化自重构机器人应对现实世界任务的设计奠定了基础。