Vision-Language Navigation (VLN) is a challenging task that requires an
embodied agent to perform action-level modality alignment, i.e., make
instruction-asked actions sequentially in complex visual environments. Most
existing VLN agents learn the instruction-path data directly and cannot
sufficiently explore action-level alignment knowledge inside the multi-modal
inputs. In this paper, we propose modAlity-aligneD Action PrompTs (ADAPT),
which provides the VLN agent with action prompts to enable the explicit
learning of action-level modality alignment to pursue successful navigation.
Specifically, an action prompt is defined as a modality-aligned pair of an
image sub-prompt and a text sub-prompt, where the former is a single-view
observation and the latter is a phrase like ''walk past the chair''. When
starting navigation, the instruction-related action prompt set is retrieved
from a pre-built action prompt base and passed through a prompt encoder to
obtain the prompt feature. Then the prompt feature is concatenated with the
original instruction feature and fed to a multi-layer transformer for action
prediction. To collect high-quality action prompts into the prompt base, we use
the Contrastive Language-Image Pretraining (CLIP) model which has powerful
cross-modality alignment ability. A modality alignment loss and a sequential
consistency loss are further introduced to enhance the alignment of the action
prompt and enforce the agent to focus on the related prompt sequentially.
Experimental results on both R2R and RxR show the superiority of ADAPT over
state-of-the-art methods.

本文提出了一种 Modality-Alignment Action Prompts (ADAPT) 方法，通过显式学习行动水平的模态对齐来实现对视觉环境下指令级操作的感知导航，并通过对高质量行动提示进行收集来提升对相关提示的对齐性。