Vision language decision making (VLDM) is a challenging multimodal task. The
agent have to understand complex human instructions and complete compositional
tasks involving environment navigation and object manipulation. However, the
long action sequences involved in VLDM make the task difficult to learn. From
an environment perspective, we find that task episodes can be divided into
fine-grained \textit{units}, each containing a navigation phase and an
interaction phase. Since the environment within a unit stays unchanged, we
propose a novel hybrid-training framework that enables active exploration in
the environment and reduces the exposure bias. Such framework leverages the
unit-grained configurations and is model-agnostic. Specifically, we design a
Unit-Transformer (UT) with an intrinsic recurrent state that maintains a
unit-scale cross-modal memory. Through extensive experiments on the TEACH
benchmark, we demonstrate that our proposed framework outperforms existing
state-of-the-art methods in terms of all evaluation metrics. Overall, our work
introduces a novel approach to tackling the VLDM task by breaking it down into
smaller, manageable units and utilizing a hybrid-training framework. By doing
so, we provide a more flexible and effective solution for multimodal decision
making.

本研究提出了一种基于 Unit-Transformer 的混合训练框架，将 Vision language decision making 任务分解为可管理的单元，并通过活跃的环境探索和内部循环状态来改进模型，最终取得优于现有方法的成果。

分解任务：基于单元级别的混合训练框架用于视觉和语言决策

Breaking Down the Task: A Unit-Grained Hybrid Training Framework for  Vision and Language Decision Making

In instruction conditioned navigation, agents interpret natural language and
their surroundings to navigate through an environment. Datasets for studying
this task typically contain pairs of these instructions and reference
trajectories. Yet, most evaluation metrics used thus far fail to properly
account for the latter, relying instead on insufficient similarity comparisons.
We address fundamental flaws in previously used metrics and show how Dynamic
Time Warping (DTW), a long known method of measuring similarity between two
time series, can be used for evaluation of navigation agents. For such, we
define the normalized Dynamic Time Warping (nDTW) metric, that softly penalizes
deviations from the reference path, is naturally sensitive to the order of the
nodes composing each path, is suited for both continuous and graph-based
evaluations, and can be efficiently calculated. Further, we define SDTW, which
constrains nDTW to only successful paths. We collect human similarity judgments
for simulated paths and find nDTW correlates better with human rankings than
all other metrics. We also demonstrate that using nDTW as a reward signal for
Reinforcement Learning navigation agents improves their performance on both the
Room-to-Room (R2R) and Room-for-Room (R4R) datasets. The R4R results in
particular highlight the superiority of SDTW over previous success-constrained
metrics.

本研究提出了一种新的度量方式 “normalized Dynamic Time Warping（nDTW）”，通过对 agent 的运动轨迹与参照路线的对比，较好地解决了以往评估方式的缺陷，该方法在各方面表现优异，且应用于强化学习导航智能体上可显著提高其性能。