In the Vision-and-Language Navigation in Continuous Environments (VLN-CE)
task, the human user guides an autonomous agent to reach a target goal via a
series of low-level actions following a textual instruction in natural
language. However, most existing methods do not address the likely case where
users may make mistakes when providing such instruction (e.g. "turn left"
instead of "turn right"). In this work, we address a novel task of Interactive
VLN in Continuous Environments (IVLN-CE), which allows the agent to interact
with the user during the VLN-CE navigation to verify any doubts regarding the
instruction errors. We propose an Interactive Instruction Error Detector and
Localizer (I2EDL) that triggers the user-agent interaction upon the detection
of instruction errors during the navigation. We leverage a pre-trained module
to detect instruction errors and pinpoint them in the instruction by
cross-referencing the textual input and past observations. In such way, the
agent is able to query the user for a timely correction, without demanding the
user's cognitive load, as we locate the probable errors to a precise part of
the instruction. We evaluate the proposed I2EDL on a dataset of instructions
containing errors, and further devise a novel metric, the Success weighted by
Interaction Number (SIN), to reflect both the navigation performance and the
interaction effectiveness. We show how the proposed method can ask focused
requests for corrections to the user, which in turn increases the navigation
success, while minimizing the interactions.

基于视觉和语言的连续环境中的导航任务中，我们提出了一种交互式指令错误检测和定位方法（I2EDL），允许智能代理在导航过程中与用户进行交互以验证指令错误，并通过准确定位错误部分，及时向用户查询纠正，从而提高导航成功率，减少交互次数。

I2EDL：交互式指令错误检测和定位

I2EDL: Interactive Instruction Error Detection and Localization

Vision-and-Language Navigation in Continuous Environments (VLN-CE) is one of
the most intuitive yet challenging embodied AI tasks. Agents are tasked to
navigate towards a target goal by executing a set of low-level actions,
following a series of natural language instructions. All VLN-CE methods in the
literature assume that language instructions are exact. However, in practice,
instructions given by humans can contain errors when describing a spatial
environment due to inaccurate memory or confusion. Current VLN-CE benchmarks do
not address this scenario, making the state-of-the-art methods in VLN-CE
fragile in the presence of erroneous instructions from human users. For the
first time, we propose a novel benchmark dataset that introduces various types
of instruction errors considering potential human causes. This benchmark
provides valuable insight into the robustness of VLN systems in continuous
environments. We observe a noticeable performance drop (up to -25%) in Success
Rate when evaluating the state-of-the-art VLN-CE methods on our benchmark.
Moreover, we formally define the task of Instruction Error Detection and
Localization, and establish an evaluation protocol on top of our benchmark
dataset. We also propose an effective method, based on a cross-modal
transformer architecture, that achieves the best performance in error detection
and localization, compared to baselines. Surprisingly, our proposed method has
revealed errors in the validation set of the two commonly used datasets for
VLN-CE, i.e., R2R-CE and RxR-CE, demonstrating the utility of our technique in
other tasks. Code and dataset will be made available upon acceptance at
this https URL

在连续环境中的视觉与语言导航 (VLN-CE) 是一项直观但具有挑战性的 AI 任务，我们提出了一个新颖的基准数据集，引入了各种类型的指令错误并考虑了潜在的人为原因，从而为连续环境中的 VLN 系统的鲁棒性提供了宝贵的见解。我们还正式定义了指令错误检测和定位的任务，并在我们的基准数据集上建立了一个评估协议，我们提出的基于跨模态变换器架构的有效方法在错误检测和定位方面表现最好。