Many Vision-and-Language Navigation (VLN) tasks have been proposed in recent
years, from room-based to object-based and indoor to outdoor. The REVERIE
(Remote Embodied Referring Expression) is interesting since it only provides
high-level instructions to the agent, which are closer to human commands in
practice. Nevertheless, this poses more challenges than other VLN tasks since
it requires agents to infer a navigation plan only based on a short
instruction. Large Language Models (LLMs) show great potential in robot action
planning by providing proper prompts. Still, this strategy has not been
explored under the REVERIE settings. There are several new challenges. For
example, the LLM should be environment-aware so that the navigation plan can be
adjusted based on the current visual observation. Moreover, the LLM planned
actions should be adaptable to the much larger and more complex REVERIE
environment. This paper proposes a March-in-Chat (MiC) model that can talk to
the LLM on the fly and plan dynamically based on a newly proposed
Room-and-Object Aware Scene Perceiver (ROASP). Our MiC model outperforms the
previous state-of-the-art by large margins by SPL and RGSPL metrics on the
REVERIE benchmark.

这篇论文提出了一种名为 March-in-Chat (MiC) 的模型，可以与大型语言模型进行交互，并基于一种新提出的会考虑房间和物体的感知模型（ROASP）进行动态规划，为 REVERIE 基准测试中的 SPL 和 RGSPL 指标提供了比以往最优模型更好的性能。

聊天中的三月：用于远程身体引用表达的交互提示

March in Chat: Interactive Prompting for Remote Embodied Referring  Expression

Humans learn from life events to form intuitions towards the understanding of
visual environments and languages. Envision that you are instructed by a
high-level instruction, "Go to the bathroom in the master bedroom and replace
the blue towel on the left wall", what would you possibly do to carry out the
task? Intuitively, we comprehend the semantics of the instruction to form an
overview of where a bathroom is and what a blue towel is in mind; then, we
navigate to the target location by consistently matching the bathroom
appearance in mind with the current scene. In this paper, we present an agent
that mimics such human behaviors. Specifically, we focus on the Remote Embodied
Visual Referring Expression in Real Indoor Environments task, called REVERIE,
where an agent is asked to correctly localize a remote target object specified
by a concise high-level natural language instruction, and propose a two-stage
training pipeline. In the first stage, we pretrain the agent with two
cross-modal alignment sub-tasks, namely the Scene Grounding task and the Object
Grounding task. The agent learns where to stop in the Scene Grounding task and
what to attend to in the Object Grounding task respectively. Then, to generate
action sequences, we propose a memory-augmented attentive action decoder to
smoothly fuse the pre-trained vision and language representations with the
agent's past memory experiences. Without bells and whistles, experimental
results show that our method outperforms previous state-of-the-art(SOTA)
significantly, demonstrating the effectiveness of our method.

本文提出了一个能够模拟人类行为的代理程序，旨在解决 REVERIE 任务，通过两个交叉模态对齐子任务的预训练阶段，即场景定位任务和对象定位任务，再结合记忆增强注意力动作解码器来生成行动序列，证明了本方法的有效性。