Understanding spatial and visual information is essential for a navigation
agent who follows natural language instructions. The current Transformer-based
VLN agents entangle the orientation and vision information, which limits the
gain from the learning of each information source. In this paper, we design a
neural agent with explicit Orientation and Vision modules. Those modules learn
to ground spatial information and landmark mentions in the instructions to the
visual environment more effectively. To strengthen the spatial reasoning and
visual perception of the agent, we design specific pre-training tasks to feed
and better utilize the corresponding modules in our final navigation model. We
evaluate our approach on both Room2room (R2R) and Room4room (R4R) datasets and
achieve the state of the art results on both benchmarks.

本文设计了一种具有显式方向和视觉模块的神经代理，通过特定的预训练任务，强化代理的空间推理和视觉感知，在 Room2room 和 Room4room 数据集上均取得了最先进的结果。

LOViS: 为视觉语言导航学习方向和视觉信号

LOViS: Learning Orientation and Visual Signals for Vision and Language Navigation

Vision-and-language navigation (VLN) is a multimodal task where an agent
follows natural language instructions and navigates in visual environments.
Multiple setups have been proposed, and researchers apply new model
architectures or training techniques to boost navigation performance. However,
there still exist non-negligible gaps between machines' performance and human
benchmarks. Moreover, the agents' inner mechanisms for navigation decisions
remain unclear. To the best of our knowledge, how the agents perceive the
multimodal input is under-studied and needs investigation. In this work, we
conduct a series of diagnostic experiments to unveil agents' focus during
navigation. Results show that indoor navigation agents refer to both object and
direction tokens when making decisions. In contrast, outdoor navigation agents
heavily rely on direction tokens and poorly understand the object tokens.
Transformer-based agents acquire a better cross-modal understanding of objects
and display strong numerical reasoning ability than non-Transformer-based
agents. When it comes to vision-and-language alignments, many models claim that
they can align object tokens with specific visual targets. We find unbalanced
attention on the vision and text input and doubt the reliability of such
cross-modal alignments.

本文研究了视觉与语言导航的多模态任务，通过一系列诊断实验揭示不同类型的导航智能体的注意力焦点和跨模态理解能力，发现 Transformer 模型相对其他模型更具有对物体的跨模态理解能力，同时对于模型声称的跨模态对齐方式存在质疑。