Vision-and-language navigation (VLN) requires an embodied agent to navigate
in realistic 3D environments using natural language instructions. Existing VLN
methods suffer from training on small-scale environments or unreasonable
path-instruction datasets, limiting the generalization to unseen environments.
There are massive house tour videos on YouTube, providing abundant real
navigation experiences and layout information. However, these videos have not
been explored for VLN before. In this paper, we propose to learn an agent from
these videos by creating a large-scale dataset which comprises reasonable
path-instruction pairs from house tour videos and pre-training the agent on it.
To achieve this, we have to tackle the challenges of automatically constructing
path-instruction pairs and exploiting real layout knowledge from raw and
unlabeled videos. To address these, we first leverage an entropy-based method
to construct the nodes of a path trajectory. Then, we propose an action-aware
generator for generating instructions from unlabeled trajectories. Last, we
devise a trajectory judgment pretext task to encourage the agent to mine the
layout knowledge. Experimental results show that our method achieves
state-of-the-art performance on two popular benchmarks (R2R and REVERIE). Code
is available at this https URL

通过利用大规模的房屋导览视频数据集，本文提出了一种视觉语言导航方法，使用自动构建的路径指令对训练的代理进行预训练，并利用视频中的布局信息来提高导航的泛化性能。通过实验结果表明，该方法在两个流行的基准测试（R2R 和 REVERIE）上取得了最先进的性能。