VLN-CE is a recently released embodied task, where AI agents need to navigate
a freely traversable environment to reach a distant target location, given
language instructions. It poses great challenges due to the huge space of
possible strategies. Driven by the belief that the ability to anticipate the
consequences of future actions is crucial for the emergence of intelligent and
interpretable planning behavior, we propose DREAMWALKER -- a world model based
VLN-CE agent. The world model is built to summarize the visual, topological,
and dynamic properties of the complicated continuous environment into a
discrete, structured, and compact representation. DREAMWALKER can simulate and
evaluate possible plans entirely in such internal abstract world, before
executing costly actions. As opposed to existing model-free VLN-CE agents
simply making greedy decisions in the real world, which easily results in
shortsighted behaviors, DREAMWALKER is able to make strategic planning through
large amounts of ``mental experiments.'' Moreover, the imagined future
scenarios reflect our agent's intention, making its decision-making process
more transparent. Extensive experiments and ablation studies on VLN-CE dataset
confirm the effectiveness of the proposed approach and outline fruitful
directions for future work.

通过构建一个基于世界模型的 VLN-CE 代理程序，DREAMWALKER 能够在内部抽象世界中模拟和评估可能的计划，实现智能导航和透明的决策过程。

DREAMWALKER：持续视觉语言导航的心智规划

DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation

We have observed significant progress in visual navigation for embodied
agents. A common assumption in studying visual navigation is that the
environments are static; this is a limiting assumption. Intelligent navigation
may involve interacting with the environment beyond just moving
forward/backward and turning left/right. Sometimes, the best way to navigate is
to push something out of the way. In this paper, we study the problem of
interactive navigation where agents learn to change the environment to navigate
more efficiently to their goals. To this end, we introduce the Neural
Interaction Engine (NIE) to explicitly predict the change in the environment
caused by the agent's actions. By modeling the changes while planning, we find
that agents exhibit significant improvements in their navigational
capabilities. More specifically, we consider two downstream tasks in the
physics-enabled, visually rich, AI2-THOR environment: (1) reaching a target
while the path to the target is blocked (2) moving an object to a target
location by pushing it. For both tasks, agents equipped with an NIE
significantly outperform agents without the understanding of the effect of the
actions indicating the benefits of our approach.

本文介绍了智能体互动导航的问题，并提出了用神经交互引擎（NIE）解决物理任务的方法，研究表明，应用 NIE 的智能体的导航能力显着提高。