In Vision-and-Language Navigation (VLN), researchers typically take an image
encoder pre-trained on ImageNet without fine-tuning on the environments that
the agent will be trained or tested on. However, the distribution shift between
the training images from ImageNet and the views in the navigation environments
may render the ImageNet pre-trained image encoder suboptimal. Therefore, in
this paper, we design a set of structure-encoding auxiliary tasks (SEA) that
leverage the data in the navigation environments to pre-train and improve the
image encoder. Specifically, we design and customize (1) 3D jigsaw, (2)
traversability prediction, and (3) instance classification to pre-train the
image encoder. Through rigorous ablations, our SEA pre-trained features are
shown to better encode structural information of the scenes, which ImageNet
pre-trained features fail to properly encode but is crucial for the target
navigation task. The SEA pre-trained features can be easily plugged into
existing VLN agents without any tuning. For example, on Test-Unseen
environments, the VLN agents combined with our SEA pre-trained features achieve
absolute success rate improvement of 12% for Speaker-Follower, 5% for
Env-Dropout, and 4% for AuxRN.

通过结构编码辅助任务 (SEA) 预先训练视觉编码器，可以提高视觉和语言导航中绝对成功率，尤其是在测试环境中，这种预训练技术可以很好地编码场景的结构信息。