Vision-and-Language Navigation in Continuous Environments (VLN-CE) is one of
the most intuitive yet challenging embodied AI tasks. Agents are tasked to
navigate towards a target goal by executing a set of low-level actions,
following a series of natural language instructions. All VLN-CE methods in the
literature assume that language instructions are exact. However, in practice,
instructions given by humans can contain errors when describing a spatial
environment due to inaccurate memory or confusion. Current VLN-CE benchmarks do
not address this scenario, making the state-of-the-art methods in VLN-CE
fragile in the presence of erroneous instructions from human users. For the
first time, we propose a novel benchmark dataset that introduces various types
of instruction errors considering potential human causes. This benchmark
provides valuable insight into the robustness of VLN systems in continuous
environments. We observe a noticeable performance drop (up to -25%) in Success
Rate when evaluating the state-of-the-art VLN-CE methods on our benchmark.
Moreover, we formally define the task of Instruction Error Detection and
Localization, and establish an evaluation protocol on top of our benchmark
dataset. We also propose an effective method, based on a cross-modal
transformer architecture, that achieves the best performance in error detection
and localization, compared to baselines. Surprisingly, our proposed method has
revealed errors in the validation set of the two commonly used datasets for
VLN-CE, i.e., R2R-CE and RxR-CE, demonstrating the utility of our technique in
other tasks. Code and dataset will be made available upon acceptance at
this https URL

在连续环境中的视觉与语言导航 (VLN-CE) 是一项直观但具有挑战性的 AI 任务，我们提出了一个新颖的基准数据集，引入了各种类型的指令错误并考虑了潜在的人为原因，从而为连续环境中的 VLN 系统的鲁棒性提供了宝贵的见解。我们还正式定义了指令错误检测和定位的任务，并在我们的基准数据集上建立了一个评估协议，我们提出的基于跨模态变换器架构的有效方法在错误检测和定位方面表现最好。

注意错误！视觉与语言导航中的指示错误检测和定位

Mind the Error! Detection and Localization of Instruction Errors in  Vision-and-Language Navigation

VLN-CE is a recently released embodied task, where AI agents need to navigate
a freely traversable environment to reach a distant target location, given
language instructions. It poses great challenges due to the huge space of
possible strategies. Driven by the belief that the ability to anticipate the
consequences of future actions is crucial for the emergence of intelligent and
interpretable planning behavior, we propose DREAMWALKER -- a world model based
VLN-CE agent. The world model is built to summarize the visual, topological,
and dynamic properties of the complicated continuous environment into a
discrete, structured, and compact representation. DREAMWALKER can simulate and
evaluate possible plans entirely in such internal abstract world, before
executing costly actions. As opposed to existing model-free VLN-CE agents
simply making greedy decisions in the real world, which easily results in
shortsighted behaviors, DREAMWALKER is able to make strategic planning through
large amounts of ``mental experiments.'' Moreover, the imagined future
scenarios reflect our agent's intention, making its decision-making process
more transparent. Extensive experiments and ablation studies on VLN-CE dataset
confirm the effectiveness of the proposed approach and outline fruitful
directions for future work.

通过构建一个基于世界模型的 VLN-CE 代理程序，DREAMWALKER 能够在内部抽象世界中模拟和评估可能的计划，实现智能导航和透明的决策过程。

DREAMWALKER：持续视觉语言导航的心智规划

DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation

Recent work in Vision-and-Language Navigation (VLN) has presented two
environmental paradigms with differing realism -- the standard VLN setting
built on topological environments where navigation is abstracted away, and the
VLN-CE setting where agents must navigate continuous 3D environments using
low-level actions. Despite sharing the high-level task and even the underlying
instruction-path data, performance on VLN-CE lags behind VLN significantly. In
this work, we explore this gap by transferring an agent from the abstract
environment of VLN to the continuous environment of VLN-CE. We find that this
sim-2-sim transfer is highly effective, improving over the prior state of the
art in VLN-CE by +12% success rate. While this demonstrates the potential for
this direction, the transfer does not fully retain the original performance of
the agent in the abstract setting. We present a sequence of experiments to
identify what differences result in performance degradation, providing clear
directions for further improvement.

这篇论文探讨了如何通过迁移学习将抽象的 VLN 任务应用到真实的 VLN-CE 任务中，提高了原先 VLN-CE 任务的成功率，但并没有完全保持抽象环境下的表现，并且通过一系列实验找出了导致性能降低的原因，并提供了进一步改进的清晰方向。