Due to the need to interact with the real world, embodied agents are required
to possess comprehensive prior knowledge, long-horizon planning capability, and
a swift response speed. Despite recent large language model (LLM) based agents
achieving promising performance, they still exhibit several limitations. For
instance, the output of LLMs is a descriptive sentence, which is ambiguous when
determining specific actions. To address these limitations, we introduce the
large auto-regressive model (LARM). LARM leverages both text and multi-view
images as input and predicts subsequent actions in an auto-regressive manner.
To train LARM, we develop a novel data format named auto-regressive node
transmission structure and assemble a corresponding dataset. Adopting a
two-phase training regimen, LARM successfully harvests enchanted equipment in
Minecraft, which demands significantly more complex decision-making chains than
the highest achievements of prior best methods. Besides, the speed of LARM is
6.8x faster.

为了解决大型语言模型机器人在特定行动中的限制，本论文提出了一种新的大型自回归模型（LARM），该模型利用文本和多视图图像作为输入并以自回归方式预测后续行动。通过采用自回归节点传输结构的新数据格式和对应的数据集训练 LARM，成功实现了在 Minecraft 中获取更复杂的装备的决策过程。此外，LARM 的速度提高了 6.8 倍。