Following language instructions to navigate in unseen environments is a
challenging task for autonomous embodied agents. With strong representation
capabilities, pretrained vision-and-language models are widely used in VLN.
However, most of them are trained on web-crawled general-purpose datasets,
which incurs a considerable domain gap when used for VLN tasks. To address the
problem, we propose a novel and model-agnostic domain-aware prompt learning
(DAP) framework. For equipping the pretrained models with specific object-level
and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost
prompt tuning paradigm to learn soft visual prompts for extracting in-domain
image semantics. Specifically, we first generate a set of in-domain image-text
pairs with the help of the CLIP model. Then we introduce soft visual prompts in
the input space of the visual encoder in a pretrained model. DAP injects
in-domain visual knowledge into the visual encoder of the pretrained model in
an efficient way. Experimental results on both R2R and REVERIE show the
superiority of DAP compared to existing state-of-the-art methods.

通过引入低成本提示调整范式来学习特定领域视觉提示，我们提出了一种新颖且模型无关的领域感知提示学习（DAP）框架，以在 VLN 任务中为预训练模型提供特定对象级和场景级跨模态对齐，并将领域内视觉知识以高效的方式注入预训练模型的视觉编码器。在 R2R 和 REVERIE 上的实验结果显示，DAP 相比现有的最先进方法具有明显的优势。