Vision-language navigation (VLN) is a challenging task due to its large
searching space in the environment. To address this problem, previous works
have proposed some methods of fine-tuning a large model that pretrained on
large-scale datasets. However, the conventional fine-tuning methods require
extra human-labeled navigation data and lack self-exploration capabilities in
environments, which hinders their generalization of unseen scenes. To improve
the ability of fast cross-domain adaptation, we propose Prompt-based
Environmental Self-exploration (ProbES), which can self-explore the
environments by sampling trajectories and automatically generates structured
instructions via a large-scale cross-modal pretrained model (CLIP). Our method
fully utilizes the knowledge learned from CLIP to build an in-domain dataset by
self-exploration without human labeling. Unlike the conventional approach of
fine-tuning, we introduce prompt-based learning to achieve fast adaptation for
language embeddings, which substantially improves the learning efficiency by
leveraging prior knowledge. By automatically synthesizing
trajectory-instruction pairs in any environment without human supervision and
efficient prompt-based learning, our model can adapt to diverse vision-language
navigation tasks, including VLN and REVERIE. Both qualitative and quantitative
results show that our ProbES significantly improves the generalization ability
of the navigation model.

本文提出了一种基于 Prompt 的自我探索方法，通过对环境进行采样来自动生成结构化的指导，从而构建了一个无需人工标注的本地数据集，并引入基于 Prompt 的学习来提高语言嵌入的学习效率，进而大大提高了视觉 - 语言导航模型的泛化能力。