Vision-and-Language Navigation (VLN) requires an agent to navigate in a
real-world environment following natural language instructions. From both the
textual and visual perspectives, we find that the relationships among the
scene, its objects,and directional clues are essential for the agent to
interpret complex instructions and correctly perceive the environment. To
capture and utilize the relationships, we propose a novel Language and Visual
Entity Relationship Graph for modelling the inter-modal relationships between
text and vision, and the intra-modal relationships among visual entities. We
propose a message passing algorithm for propagating information between
language elements and visual entities in the graph, which we then combine to
determine the next action to take. Experiments show that by taking advantage of
the relationships we are able to improve over state-of-the-art. On the
Room-to-Room (R2R) benchmark, our method achieves the new best performance on
the test unseen split with success rate weighted by path length (SPL) of 52%.
On the Room-for-Room (R4R) dataset, our method significantly improves the
previous best from 13% to 34% on the success weighted by normalized dynamic
time warping (SDTW). Code is available at:
this https URL

通过提出一种语言和视觉实体关系图及信息传递算法，成功解决了视觉和语言的跨模态关系，提升了在 Room-to-Room 和 Room-for-Room 测试数据集的表现。

语言和视觉实体关系图用于智能体导航

Language and Visual Entity Relationship Graph for Agent Navigation

Advances in learning and representations have reinvigorated work that
connects language to other modalities. A particularly exciting direction is
Vision-and-Language Navigation(VLN), in which agents interpret natural language
instructions and visual scenes to move through environments and reach goals.
Despite recent progress, current research leaves unclear how much of a role
language understanding plays in this task, especially because dominant
evaluation metrics have focused on goal completion rather than the sequence of
actions corresponding to the instructions. Here, we highlight shortcomings of
current metrics for the Room-to-Room dataset (Anderson et al.,2018b) and
propose a new metric, Coverage weighted by Length Score (CLS). We also show
that the existing paths in the dataset are not ideal for evaluating instruction
following because they are direct-to-goal shortest paths. We join existing
short paths to form more challenging extended paths to create a new data set,
Room-for-Room (R4R). Using R4R and CLS, we show that agents that receive
rewards for instruction fidelity outperform agents that focus on goal
completion.

通过对 Room-to-Room 数据集的评估提出新的评价指标 CLS，并且创建了新数据集 Room-for-Room (R4R) 用于指导指令跟踪的任务，同时通过比较与基准系统，表明重视指令的代理程序优于重视目标完成的代理程序。