Vision-and-Language Navigation (VLN) is a task that an agent is required to
follow a language instruction to navigate to the goal position, which relies on
the ongoing interactions with the environment during moving. Recent
Transformer-based VLN methods have made great progress benefiting from the
direct connections between visual observations and the language instruction via
the multimodal cross-attention mechanism. However, these methods usually
represent temporal context as a fixed-length vector by using an LSTM decoder or
using manually designed hidden states to build a recurrent Transformer.
Considering a single fixed-length vector is often insufficient to capture
long-term temporal context, in this paper, we introduce Multimodal Transformer
with Variable-length Memory (MTVM) for visually-grounded natural language
navigation by modelling the temporal context explicitly. Specifically, MTVM
enables the agent to keep track of the navigation trajectory by directly
storing previous activations in a memory bank. To further boost the
performance, we propose a memory-aware consistency loss to help learn a better
joint representation of temporal context with random masked instructions. We
evaluate MTVM on popular R2R and CVDN datasets, and our model improves Success
Rate on R2R unseen validation and test set by 2% each, and reduce Goal Process
by 1.6m on CVDN test set.

此篇研究提出了一种基于视觉和自然语言导航的多模态变长记忆转换器（MTVM）方法，用于模拟时间上下文，通过在存储器库中直接存储先前激活来跟踪导航轨迹，并提出了一种内存感知一致性损失来提高模型的性能。通过在 R2R 和 CVDN 数据集上的评估，该模型在验证集和测试集的成功率分别提高了 2％且将 CVDN 测试集的目标处理减少了 1.6m。