Vision language decision making (VLDM) is a challenging multimodal task. The
agent have to understand complex human instructions and complete compositional
tasks involving environment navigation and object manipulation. However, the
long action sequences involved in VLDM make the task difficult to learn. From
an environment perspective, we find that task episodes can be divided into
fine-grained \textit{units}, each containing a navigation phase and an
interaction phase. Since the environment within a unit stays unchanged, we
propose a novel hybrid-training framework that enables active exploration in
the environment and reduces the exposure bias. Such framework leverages the
unit-grained configurations and is model-agnostic. Specifically, we design a
Unit-Transformer (UT) with an intrinsic recurrent state that maintains a
unit-scale cross-modal memory. Through extensive experiments on the TEACH
benchmark, we demonstrate that our proposed framework outperforms existing
state-of-the-art methods in terms of all evaluation metrics. Overall, our work
introduces a novel approach to tackling the VLDM task by breaking it down into
smaller, manageable units and utilizing a hybrid-training framework. By doing
so, we provide a more flexible and effective solution for multimodal decision
making.

本研究提出了一种基于 Unit-Transformer 的混合训练框架，将 Vision language decision making 任务分解为可管理的单元，并通过活跃的环境探索和内部循环状态来改进模型，最终取得优于现有方法的成果。