Navigation instruction generation, which requires embodied agents to describe
the navigation routes, has been of great interest in robotics and
human-computer interaction. Existing studies directly map the sequence of 2D
perspective observations to route descriptions. Though straightforward, they
overlook the geometric information and object semantics of the 3D environment.
To address these challenges, we propose BEVInstructor, which incorporates
Bird's Eye View (BEV) features into Multi-Modal Large Language Models (MLLMs)
for instruction generation. Specifically, BEVInstructor constructs a
PerspectiveBEVVisual Encoder for the comprehension of 3D environments through
fusing BEV and perspective features. To leverage the powerful language
capabilities of MLLMs, the fused representations are used as visual prompts for
MLLMs, and perspective-BEV prompt tuning is proposed for parameter-efficient
updating. Based on the perspective-BEV prompts, BEVInstructor further adopts an
instance-guided iterative refinement pipeline, which improves the instructions
in a progressive manner. BEVInstructor achieves impressive performance across
diverse datasets (i.e., R2R, REVERIE, and UrbanWalk).

BEVInstructor 提出了一种将 Bird's Eye View（BEV）特征与 Multi-Modal Large Language Models（MLLMs）相结合的导航指令生成方法，通过融合 BEV 和透视特征构建 PerspectiveBEVVisual Encoder 以提高对 3D 环境的理解能力，并采用基于透视 - BEV 提示的参数高效更新和实例引导的迭代细化流程，以逐步改进指令的效果在不同数据集上取得了令人瞩目的成果（即 R2R、REVERIE 和 UrbanWalk）。

使用 BEV 感知和大型语言模型生成导航指令

Navigation Instruction Generation with BEV Perception and Large Language  Models

We introduce a novel speaker model \textsc{Kefa} for navigation instruction
generation. The existing speaker models in Vision-and-Language Navigation
suffer from the large domain gap of vision features between different
environments and insufficient temporal grounding capability. To address the
challenges, we propose a Knowledge Refinement Module to enhance the feature
representation with external knowledge facts, and an Adaptive Temporal
Alignment method to enforce fine-grained alignment between the generated
instructions and the observation sequences. Moreover, we propose a new metric
SPICE-D for navigation instruction evaluation, which is aware of the
correctness of direction phrases. The experimental results on R2R and UrbanWalk
datasets show that the proposed KEFA speaker achieves state-of-the-art
instruction generation performance for both indoor and outdoor scenes.

我们引入了一种新颖的导航指令生成的说话者模型 Kefa。我们的模型通过知识细化模块来增强特征表示，并通过自适应时间对齐方法来确保生成的指令与观测序列的细粒度对齐。此外，我们提出了一种新的导航指令评估度量 SPICE-D，该度量关注方向短语的正确性。在 R2R 和 UrbanWalk 数据集上的实验证明了我们提出的 KEFA 说话者模型在室内外场景的指令生成性能上达到了最先进水平。