Embodied AI focuses on the study and development of intelligent systems that
possess a physical or virtual embodiment (i.e. robots) and are able to
dynamically interact with their environment. Memory and control are the two
essential parts of an embodied system and usually require separate frameworks
to model each of them. In this paper, we propose a novel and generalizable
framework called LLM-Brain: using Large-scale Language Model as a robotic brain
to unify egocentric memory and control. The LLM-Brain framework integrates
multiple multimodal language models for robotic tasks, utilizing a zero-shot
learning approach. All components within LLM-Brain communicate using natural
language in closed-loop multi-round dialogues that encompass perception,
planning, control, and memory. The core of the system is an embodied LLM to
maintain egocentric memory and control the robot. We demonstrate LLM-Brain by
examining two downstream tasks: active exploration and embodied question
answering. The active exploration tasks require the robot to extensively
explore an unknown environment within a limited number of actions. Meanwhile,
the embodied question answering tasks necessitate that the robot answers
questions based on observations acquired during prior explorations.

本文提出了一个名为 LLM-Brain 的新型机器人控制框架，旨在利用大规模语言模型作为机器人的大脑，统一身体自我感知记忆和控制，并利用多模态语言模型来实现机器人任务的零样本学习。该框架可以通过多轮对话实现感知、规划、控制和记忆等任务，并通过两个下游任务（主动探测和实体问题回答）的实验来验证其有效性。