We present OmniJARVIS, a novel Vision-Language-Action (VLA) model for
open-world instruction-following agents in open-world Minecraft. Compared to
prior works that either emit textual goals to separate controllers or produce
the control command directly, OmniJARVIS seeks a different path to ensure both
strong reasoning and efficient decision-making capabilities via unified
tokenization of multimodal interaction data. First, we introduce a
self-supervised approach to learn a behavior encoder that produces discretized
tokens for behavior trajectories $\tau$ = {$o_0$, $a_0$, $\dots$} and an
imitation learning (IL) policy decoder conditioned on these tokens. These
additional behavior tokens will be augmented to the vocabulary of pretrained
Multimodal Language Models (MLMs). With this encoder, we then pack long-term
multimodal interactions involving task instructions, memories, thoughts,
observations, textual responses, behavior trajectories, etc. into unified token
sequences and model them with autoregressive transformers. Thanks to the
semantically meaningful behavior tokens, the resulting VLA model, OmniJARVIS,
can reason (by producing chain-of-thoughts), plan, answer questions, and act
(by producing behavior tokens for the IL policy decoder). OmniJARVIS
demonstrates excellent performances on a comprehensive collection of atomic,
programmatic, and open-ended tasks in open-world Minecraft. Our analysis
further unveils the crucial design principles in interaction data formation,
unified tokenization, and its scaling potentials.

OmniJARVIS 是一个新颖的视觉 - 语言 - 动作（VLA）模型，用于开放世界 Minecraft 中的指令跟随代理。通过对多模态交互数据进行统一分词，OmniJARVIS 能够具备强大的推理和高效的决策能力，并在开放世界 Minecraft 中的多种任务上表现出优异的性能。

OmniJARVIS: 统一的视觉 - 语言 - 行为标记使得开放世界指导跟随代理

OmniJARVIS: Unified Vision-Language-Action Tokenization Enables  Open-World Instruction Following Agents

Despite advancements in Large Language Models (LLMs) and Large Multimodal
Models (LMMs), their integration into language-grounded, human-like embodied
agents remains incomplete, hindering complex real-life task performance in
physical environments. Existing integrations often feature limited open
sourcing, challenging collective progress in this field. We introduce LEGENT,
an open, scalable platform for developing embodied agents using LLMs and LMMs.
LEGENT offers a dual approach: a rich, interactive 3D environment with
communicable and actionable agents, paired with a user-friendly interface, and
a sophisticated data generation pipeline utilizing advanced algorithms to
exploit supervision from simulated worlds at scale. In our experiments, an
embryonic vision-language-action model trained on LEGENT-generated data
surpasses GPT-4V in embodied tasks, showcasing promising generalization
capabilities.

通过使用 LEGENT 生成的数据，我们训练的视觉 - 语言 - 动作模型在具体任务中超越了 GPT-4V，展示了令人期待的泛化能力。