It is a long-lasting goal to design a generalist-embodied agent that can
follow diverse instructions in human-like ways. However, existing approaches
often fail to steadily follow instructions due to difficulties in understanding
abstract and sequential natural language instructions. To this end, we
introduce MineDreamer, an open-ended embodied agent built upon the challenging
Minecraft simulator with an innovative paradigm that enhances
instruction-following ability in low-level control signal generation.
Specifically, MineDreamer is developed on top of recent advances in Multimodal
Large Language Models (MLLMs) and diffusion models, and we employ a
Chain-of-Imagination (CoI) mechanism to envision the step-by-step process of
executing instructions and translating imaginations into more precise visual
prompts tailored to the current state; subsequently, the agent generates
keyboard-and-mouse actions to efficiently achieve these imaginations, steadily
following the instructions at each step. Extensive experiments demonstrate that
MineDreamer follows single and multi-step instructions steadily, significantly
outperforming the best generalist agent baseline and nearly doubling its
performance. Moreover, qualitative analysis of the agent's imaginative ability
reveals its generalization and comprehension of the open world.

设计一种通用 - 具身化代理，通过使用多模态大型语言模型和扩散模型的先进技术，并利用想象链机制，实现低级控制信号生成中的指令跟随能力的增强。MineDreamer 在 Minecraft 模拟器上开发，能够稳定地按照单步和多步指令执行，胜过通用代理基准，并提高了其性能，并通过定性分析显示其对开放世界的泛化和理解能力。