We introduce a novel speaker model \textsc{Kefa} for navigation instruction
generation. The existing speaker models in Vision-and-Language Navigation
suffer from the large domain gap of vision features between different
environments and insufficient temporal grounding capability. To address the
challenges, we propose a Knowledge Refinement Module to enhance the feature
representation with external knowledge facts, and an Adaptive Temporal
Alignment method to enforce fine-grained alignment between the generated
instructions and the observation sequences. Moreover, we propose a new metric
SPICE-D for navigation instruction evaluation, which is aware of the
correctness of direction phrases. The experimental results on R2R and UrbanWalk
datasets show that the proposed KEFA speaker achieves state-of-the-art
instruction generation performance for both indoor and outdoor scenes.

我们引入了一种新颖的导航指令生成的说话者模型 Kefa。我们的模型通过知识细化模块来增强特征表示，并通过自适应时间对齐方法来确保生成的指令与观测序列的细粒度对齐。此外，我们提出了一种新的导航指令评估度量 SPICE-D，该度量关注方向短语的正确性。在 R2R 和 UrbanWalk 数据集上的实验证明了我们提出的 KEFA 说话者模型在室内外场景的指令生成性能上达到了最先进水平。

Kefa：基于知识增强和细粒度对齐技术的导航指令生成说话人

Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for  Navigation Instruction Generation

Vision-and-language navigation (VLN) is a crucial but challenging cross-modal
navigation task. One powerful technique to enhance the generalization
performance in VLN is the use of an independent speaker model to provide pseudo
instructions for data augmentation. However, current speaker models based on
Long-Short Term Memory (LSTM) lack the ability to attend to features relevant
at different locations and time steps. To address this, we propose a novel
progress-aware spatio-temporal transformer speaker (PASTS) model that uses the
transformer as the core of the network. PASTS uses a spatio-temporal encoder to
fuse panoramic representations and encode intermediate connections through
steps. Besides, to avoid the misalignment problem that could result in
incorrect supervision, a speaker progress monitor (SPM) is proposed to enable
the model to estimate the progress of instruction generation and facilitate
more fine-grained caption results. Additionally, a multifeature dropout (MFD)
strategy is introduced to alleviate overfitting. The proposed PASTS is flexible
to be combined with existing VLN models. The experimental results demonstrate
that PASTS outperforms all existing speaker models and successfully improves
the performance of previous VLN models, achieving state-of-the-art performance
on the standard Room-to-Room (R2R) dataset.

通过使用进度感知的时空转换演讲者模型（PASTS），该研究使用 transformer 作为网络的核心，提出了一种改进视觉和语言导航（VLN）性能的技术，该模型具有灵活性，可与现有的 VLN 模型结合使用，实验结果显示，PASTS 优于所有现有的演讲者模型，并成功提高了前期 VLN 模型的性能，在标准 R2R 数据集上实现了最先进的性能。

PASTS: 基于进展感知的时空变换演说家，用于视觉语言导航

PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For  Vision-and-Language Navigation

Natural language is an intuitive and expressive way to communicate reward
information to autonomous agents. It encompasses everything from concrete
instructions to abstract descriptions of the world. Despite this, natural
language is often challenging to learn from: it is difficult for machine
learning methods to make appropriate inferences from such a wide range of
input. This paper proposes a generalization of reward design as a unifying
principle to ground linguistic communication: speakers choose utterances to
maximize expected rewards from the listener's future behaviors. We first extend
reward design to incorporate reasoning about unknown future states in a linear
bandit setting. We then define a speaker model which chooses utterances
according to this objective. Simulations show that short-horizon speakers
(reasoning primarily about a single, known state) tend to use instructions,
while long-horizon speakers (reasoning primarily about unknown, future states)
tend to describe the reward function. We then define a pragmatic listener which
performs inverse reward design by jointly inferring the speaker's latent
horizon and rewards. Our findings suggest that this extension of reward design
to linguistic communication, including the notion of a latent speaker horizon,
is a promising direction for achieving more robust alignment outcomes from
natural language supervision.

本文提出了奖励设计的推广原则作为语言交流的基础，包括概括未知未来状态的线性赌博设置、演讲者模型和实用主义听众对演讲者潜在视角和奖励的联合推断。