Outdoor Vision-and-Language Navigation (VLN) requires an agent to navigate
through realistic 3D outdoor environments based on natural language
instructions. The performance of existing VLN methods is limited by
insufficient diversity in navigation environments and limited training data. To
address these issues, we propose VLN-Video, which utilizes the diverse outdoor
environments present in driving videos in multiple cities in the U.S. augmented
with automatically generated navigation instructions and actions to improve
outdoor VLN performance. VLN-Video combines the best of intuitive classical
approaches and modern deep learning techniques, using template infilling to
generate grounded navigation instructions, combined with an image rotation
similarity-based navigation action predictor to obtain VLN style data from
driving videos for pretraining deep learning VLN models. We pre-train the model
on the Touchdown dataset and our video-augmented dataset created from driving
videos with three proxy tasks: Masked Language Modeling, Instruction and
Trajectory Matching, and Next Action Prediction, so as to learn
temporally-aware and visually-aligned instruction representations. The learned
instruction representation is adapted to the state-of-the-art navigator when
fine-tuning on the Touchdown dataset. Empirical results demonstrate that
VLN-Video significantly outperforms previous state-of-the-art models by 2.1% in
task completion rate, achieving a new state-of-the-art on the Touchdown
dataset.

提出了 VLN-Video 方法来改善室外视觉与语言导航的性能，利用驾驶视频中多个城市的多样化室外环境和自动生成的导航指令和行动，通过结合传统直觉方法和现代深度学习技术进行预训练和微调，实现了在 Touchdown 数据集上 2.1％的任务完成率的显著提升，达到了最新的技术水平。

VLN-Video：利用行车视频进行户外视觉语言导航

VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language  Navigation

In Vision-and-Language Navigation (VLN), researchers typically take an image
encoder pre-trained on ImageNet without fine-tuning on the environments that
the agent will be trained or tested on. However, the distribution shift between
the training images from ImageNet and the views in the navigation environments
may render the ImageNet pre-trained image encoder suboptimal. Therefore, in
this paper, we design a set of structure-encoding auxiliary tasks (SEA) that
leverage the data in the navigation environments to pre-train and improve the
image encoder. Specifically, we design and customize (1) 3D jigsaw, (2)
traversability prediction, and (3) instance classification to pre-train the
image encoder. Through rigorous ablations, our SEA pre-trained features are
shown to better encode structural information of the scenes, which ImageNet
pre-trained features fail to properly encode but is crucial for the target
navigation task. The SEA pre-trained features can be easily plugged into
existing VLN agents without any tuning. For example, on Test-Unseen
environments, the VLN agents combined with our SEA pre-trained features achieve
absolute success rate improvement of 12% for Speaker-Follower, 5% for
Env-Dropout, and 4% for AuxRN.

通过结构编码辅助任务 (SEA) 预先训练视觉编码器，可以提高视觉和语言导航中绝对成功率，尤其是在测试环境中，这种预训练技术可以很好地编码场景的结构信息。