Existing work on vision and language navigation mainly relies on
navigation-related losses to establish the connection between vision and
language modalities, neglecting aspects of helping the navigation agent build a
deep understanding of the visual environment. In our work, we provide indirect
supervision to the navigation agent through a hint generator that provides
detailed visual descriptions. The hint generator assists the navigation agent
in developing a global understanding of the visual environment. It directs the
agent's attention toward related navigation details, including the relevant
sub-instruction, potential challenges in recognition and ambiguities in
grounding, and the targeted viewpoint description. To train the hint generator,
we construct a synthetic dataset based on landmarks in the instructions and
visible and distinctive objects in the visual environment. We evaluate our
method on the R2R and R4R datasets and achieve state-of-the-art on several
metrics. The experimental results demonstrate that generating hints not only
enhances the navigation performance but also helps improve the interpretability
of the agent's actions.

导航和语言导航的现有研究主要依赖于导航相关的损失，以建立视觉和语言模态之间的联系，忽视了帮助导航代理机构建立对视觉环境的深入理解的方面。在我们的工作中，我们通过提示生成器为导航代理提供间接监督，提供详细的视觉描述。提示生成器帮助导航代理机构开发对视觉环境的整体理解，指导代理机构注意相关的导航细节，包括相关的子指导、识别的潜在挑战和限制、以及目标视点描述。为了训练提示生成器，我们根据指导中的地标和可见的与众不同的物体构建了一个合成数据集。我们在 R2R 和 R4R 数据集上评估了我们的方法，并在几个度量标准上取得了最先进的结果。实验结果表明，生成提示不仅提高了导航性能，还有助于提高代理机构行动的可解释性。

NavHint: 具备提示生成器的视觉和语言导航智能体

NavHint: Vision and Language Navigation Agent with a Hint Generator

Recently, visual-language navigation (VLN) -- entailing robot agents to
follow navigation instructions -- has shown great advance. However, existing
literature put most emphasis on interpreting instructions into actions, only
delivering "dumb" wayfinding agents. In this article, we devise LANA, a
language-capable navigation agent which is able to not only execute
human-written navigation commands, but also provide route descriptions to
humans. This is achieved by simultaneously learning instruction following and
generation with only one single model. More specifically, two encoders,
respectively for route and language encoding, are built and shared by two
decoders, respectively, for action prediction and instruction generation, so as
to exploit cross-task knowledge and capture task-specific characteristics.
Throughout pretraining and fine-tuning, both instruction following and
generation are set as optimization objectives. We empirically verify that,
compared with recent advanced task-specific solutions, LANA attains better
performances on both instruction following and route description, with nearly
half complexity. In addition, endowed with language generation capability, LANA
can explain to humans its behaviors and assist human's wayfinding. This work is
expected to foster future efforts towards building more trustworthy and
socially-intelligent navigation robots.

文章介绍了 LANA，一种同时具有导航指令遵循和生成能力的艺术导航机器人，能够执行人类编写的导航命令并向人类提供路线说明。

Lana: 一款具备语言能力的导航器，用于指令遵循与生成

Lana: A Language-Capable Navigator for Instruction Following and  Generation

Language understanding is essential for the navigation agent to follow
instructions. We observe two kinds of issues in the instructions that can make
the navigation task challenging: 1. The mentioned landmarks are not
recognizable by the navigation agent due to the different vision abilities of
the instructor and the modeled agent. 2. The mentioned landmarks are applicable
to multiple targets, thus not distinctive for selecting the target among the
candidate viewpoints. To deal with these issues, we design a translator module
for the navigation agent to convert the original instructions into
easy-to-follow sub-instruction representations at each step. The translator
needs to focus on the recognizable and distinctive landmarks based on the
agent's visual abilities and the observed visual environment. To achieve this
goal, we create a new synthetic sub-instruction dataset and design specific
tasks to train the translator and the navigation agent. We evaluate our
approach on Room2Room~(R2R), Room4room~(R4R), and Room2Room Last (R2R-Last)
datasets and achieve state-of-the-art results on multiple benchmarks.

通过设计一个翻译模块，将原有语句转化为易于理解的子语句，该模块针对航行代理的视觉能力和环境观察结果，聚焦于可识别和有区分性的地标，通过训练翻译器和导航代理的特定任务，达到了在多个基准测试中取得最新颖成果的目标。