Vision-and-Language Navigation (VLN) is a challenging task in which an agent
needs to follow a language-specified path to reach a target destination. The
goal gets even harder as the actions available to the agent get simpler and
move towards low-level, atomic interactions with the environment. This setting
takes the name of low-level VLN. In this paper, we strive for the creation of
an agent able to tackle three key issues: multi-modality, long-term
dependencies, and adaptability towards different locomotive settings. To that
end, we devise "Perceive, Transform, and Act" (PTA): a fully-attentive VLN
architecture that leaves the recurrent approach behind and the first
Transformer-like architecture incorporating three different modalities -
natural language, images, and low-level actions for the agent control. In
particular, we adopt an early fusion strategy to merge lingual and visual
information efficiently in our encoder. We then propose to refine the decoding
phase with a late fusion extension between the agent's history of actions and
the perceptual modalities. We experimentally validate our model on two
datasets: PTA achieves promising results in low-level VLN on R2R and achieves
good performance in the recently proposed R4R benchmark. Our code is publicly
available at this https URL

本文提出了一种基于全注意力机制的低层 VLN 架构，用于实现跨多个模态的长期路径规划，其中包括自然语言、图像和代理人的低级行为控制，实验结果表明其在两个数据集上均获得了良好的性能。