Interactive and embodied tasks pose at least two fundamental challenges to
existing Vision & Language (VL) models, including 1) grounding language in
trajectories of actions and observations, and 2) referential disambiguation. To
tackle these challenges, we propose an Embodied MultiModal Agent (EMMA): a
unified encoder-decoder model that reasons over images and trajectories, and
casts action prediction as multimodal text generation. By unifying all tasks as
text generation, EMMA learns a language of actions which facilitates transfer
across tasks. Different to previous modular approaches with independently
trained components, we use a single multitask model where each task contributes
to goal completion. EMMA performs on par with similar models on several VL
benchmarks and sets a new state-of-the-art performance (36.81% success rate) on
the Dialog-guided Task Completion (DTC), a benchmark to evaluate dialog-guided
agents in the Alexa Arena

我们提出了一种统一的编码 - 解码模型，称为 Embodied MultiModal Agent (EMMA)，可以处理交互和具身任务，其中每个任务对目标完成做出贡献，取得了与类似模型相当的性能并在 Dialog-guided Task Completion (DTC) 中取得了新的最佳结果（36.81% 的成功率）用于评估 Alexa Arena 中的对话引导代理。