Vision and Language Navigation (VLN) requires an agent to navigate to a
target location by following natural language instructions. Most of existing
works represent a navigation candidate by the feature of the corresponding
single view where the candidate lies in. However, an instruction may mention
landmarks out of the single view as references, which might lead to failures of
textual-visual matching of existing methods. In this work, we propose a
multi-module Neighbor-View Enhanced Model (NvEM) to adaptively incorporate
visual contexts from neighbor views for better textual-visual matching.
Specifically, our NvEM utilizes a subject module and a reference module to
collect contexts from neighbor views. The subject module fuses neighbor views
at a global level, and the reference module fuses neighbor objects at a local
level. Subjects and references are adaptively determined via attention
me'chanisms. Our model also includes an action module to utilize the strong
orientation guidance (e.g., "turn left") in instructions. Each module predicts
navigation action separately and their weighted sum is used for predicting the
final action. Extensive experimental results demonstrate the effectiveness of
the proposed method on the R2R and R4R benchmarks against several
state-of-the-art navigators, and NvEM even beats some pre-training ones. Our
code is available at this https URL

本文提出了一种新的多模块的 Neighbor-View Enhanced Model (NvEM) 方法，通过利用相邻视图的视觉上下文信息和注意机制应对视觉和文本匹配不足的问题，并采用动作模块进行操纵指导，通过实验在 R2R 和 R4R 基准测试中展示了其效果的优越性。