Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require
machine agents to interpret natural language instructions and learn to act in
visually realistic environments to achieve navigation goals. The overall task
requires competence in several perception problems: successful agents combine
spatio-temporal, vision and language understanding to produce appropriate
action sequences. Our approach adapts pre-trained vision and language
representations to relevant in-domain tasks making them more effective for VLN.
Specifically, the representations are adapted to solve both a cross-modal
sequence alignment and sequence coherence task. In the sequence alignment task,
the model determines whether an instruction corresponds to a sequence of visual
frames. In the sequence coherence task, the model determines whether the
perceptual sequences are predictive sequentially in the instruction-conditioned
latent space. By transferring the domain-adapted representations, we improve
competitive agents in R2R as measured by the success rate weighted by path
length (SPL) metric.

我们的方法将预先训练的视觉和语言表示技术适应于相关领域的任务，通过在跨模态序列对齐和序列一致性任务中适应这些表示技术，从而提高了在 Room-to-Room（R2R） 带权路径成功率（SPL）度量方面的性能。