We study the task of zero-shot vision-and-language navigation (ZS-VLN), a
practical yet challenging problem in which an agent learns to navigate
following a path described by language instructions without requiring any
path-instruction annotation data. Normally, the instructions have complex
grammatical structures and often contain various action descriptions (e.g.,
"proceed beyond", "depart from"). How to correctly understand and execute these
action demands is a critical problem, and the absence of annotated data makes
it even more challenging. Note that a well-educated human being can easily
understand path instructions without the need for any special training. In this
paper, we propose an action-aware zero-shot VLN method ($A^2$Nav) by exploiting
the vision-and-language ability of foundation models. Specifically, the
proposed method consists of an instruction parser and an action-aware
navigation policy. The instruction parser utilizes the advanced reasoning
ability of large language models (e.g., GPT-3) to decompose complex navigation
instructions into a sequence of action-specific object navigation sub-tasks.
Each sub-task requires the agent to localize the object and navigate to a
specific goal position according to the associated action demand. To accomplish
these sub-tasks, an action-aware navigation policy is learned from freely
collected action-specific datasets that reveal distinct characteristics of each
action demand. We use the learned navigation policy for executing sub-tasks
sequentially to follow the navigation instruction. Extensive experiments show
$A^2$Nav achieves promising ZS-VLN performance and even surpasses the
supervised learning methods on R2R-Habitat and RxR-Habitat datasets.

我们提出了一种基于动作感知的零样本图像与语言导航（ZS-VLN）方法（$A^2$Nav），通过利用基础模型的视觉和语言能力，将复杂的导航指令分解为一系列具有特定动作要求的对象导航子任务，然后学习一个由已收集到的具有不同特征的动作数据集构建的动作感知导航策略，以便按顺序执行这些子任务，从而实现导航指令的完整执行。实验证明，$A^2$Nav 在零样本图像与语言导航方面具有很好的性能，并且在 R2R-Habitat 和 RxR-Habitat 数据集上甚至超过了监督学习方法。