Humans use their knowledge of common house layouts obtained from previous
experiences to predict nearby rooms while navigating in new environments. This
greatly helps them navigate previously unseen environments and locate their
target room. To provide layout prior knowledge to navigational agents based on
common human living spaces, we propose WIN (\textit{W}hat \textit{I}s
\textit{N}ear), a commonsense learning model for Vision Language Navigation
(VLN) tasks. VLN requires an agent to traverse indoor environments based on
descriptive navigational instructions. Unlike existing layout learning works,
WIN predicts the local neighborhood map based on prior knowledge of living
spaces and current observation, operating on an imagined global map of the
entire environment. The model infers neighborhood regions based on visual cues
of current observations, navigational history, and layout common sense. We show
that local-global planning based on locality knowledge and predicting the
indoor layout allows the agent to efficiently select the appropriate action.
Specifically, we devised a cross-modal transformer that utilizes this locality
prior for decision-making in addition to visual inputs and instructions.
Experimental results show that locality learning using WIN provides better
generalizability compared to classical VLN agents in unseen environments. Our
model performs favorably on standard VLN metrics, with Success Rate 68\% and
Success weighted by Path Length 63\% in unseen environments.

WIN 是一个用于视觉语言导航任务的常识学习模型，它基于以前的经验，预测未知环境中的相邻房间，并通过局部 - 全局规划和本地性学习提供更好的智能导航性能。

室内生活环境中增强机器人视觉 - 语言 - 导航能力的房间位置学习

What Is Near?: Room Locality Learning for Enhanced Robot  Vision-Language-Navigation in Indoor Living Environments

Vision language navigation is the task that requires an agent to navigate
through a 3D environment based on natural language instructions. One key
challenge in this task is to ground instructions with the current visual
information that the agent perceives. Most of the existing work employs soft
attention over individual words to locate the instruction required for the next
action. However, different words have different functions in a sentence (e.g.,
modifiers convey attributes, verbs convey actions). Syntax information like
dependencies and phrase structures can aid the agent to locate important parts
of the instruction. Hence, in this paper, we propose a navigation agent that
utilizes syntax information derived from a dependency tree to enhance alignment
between the instruction and the current visual scenes. Empirically, our agent
outperforms the baseline model that does not use syntax information on the
Room-to-Room dataset, especially in the unseen environment. Besides, our agent
achieves the new state-of-the-art on Room-Across-Room dataset, which contains
instructions in 3 languages (English, Hindi, and Telugu). We also show that our
agent is better at aligning instructions with the current visual information
via qualitative visualizations. Code and models:
this https URL

本文提出了一种新的导航代理程序，利用依赖树派生的句法信息增强指令和当前视觉场景之间的对齐，显著提高了在 Room-to-Room 数据集上的性能，同时在 Room-Across-Room 数据集上取得了新的最高性能水平。