Recent advances in Iterative Vision-and-Language Navigation (IVLN) introduce
a more meaningful and practical paradigm of VLN by maintaining the agent's
memory across tours of scenes. Although the long-term memory aligns better with
the persistent nature of the VLN task, it poses more challenges on how to
utilize the highly unstructured navigation memory with extremely sparse
supervision. Towards this end, we propose OVER-NAV, which aims to go over and
beyond the current arts of IVLN techniques. In particular, we propose to
incorporate LLMs and open-vocabulary detectors to distill key information and
establish correspondence between multi-modal signals. Such a mechanism
introduces reliable cross-modal supervision and enables on-the-fly
generalization to unseen scenes without the need of extra annotation and
re-training. To fully exploit the interpreted navigation data, we further
introduce a structured representation, coded Omnigraph, to effectively
integrate multi-modal information along the tour. Accompanied with a novel
omnigraph fusion mechanism, OVER-NAV is able to extract the most relevant
knowledge from omnigraph for a more accurate navigating action. In addition,
OVER-NAV seamlessly supports both discrete and continuous environments under a
unified framework. We demonstrate the superiority of OVER-NAV in extensive
experiments.

通过在迭代视觉语言导航（IVLN）技术中引入长期记忆，OVER-NAV 综合使用 LLMs 和开放词汇探测器来提炼关键信息、建立多模式信号之间的对应关系，并引入结构化表示 Omnigraph 和新颖的 Omnigraph 融合机制，以从 Omnigraph 中提取最相关的知识以实现更准确的导航操作。同时，OVER-NAV 在离散和连续环境下都能无缝支持，大量的实验证明了其卓越性能。

超导 航：通过 开放 词汇 检测 和 结构 化 表示 提升 迭代 视觉语言 导航

OVER-NAV: Elevating Iterative Vision-and-Language Navigation with  Open-Vocabulary Detection and StructurEd Representation

We present Iterative Vision-and-Language Navigation (IVLN), a paradigm for
evaluating language-guided agents navigating in a persistent environment over
time. Existing Vision-and-Language Navigation (VLN) benchmarks erase the
agent's memory at the beginning of every episode, testing the ability to
perform cold-start navigation with no prior information. However, deployed
robots occupy the same environment for long periods of time. The IVLN paradigm
addresses this disparity by training and evaluating VLN agents that maintain
memory across tours of scenes that consist of up to 100 ordered
instruction-following Room-to-Room (R2R) episodes, each defined by an
individual language instruction and a target path. We present discrete and
continuous Iterative Room-to-Room (IR2R) benchmarks comprising about 400 tours
each in 80 indoor scenes. We find that extending the implicit memory of
high-performing transformer VLN agents is not sufficient for IVLN, but agents
that build maps can benefit from environment persistence, motivating a renewed
focus on map-building agents in VLN.

本研究提出了一种名为 IVLN（迭代视觉语言导航）的新范例用于评估自然语言引导的智能体在具有持久环境的不同场景中的导航表现。