Most existing works solving Room-to-Room VLN problem only utilize RGB images
and do not consider local context around candidate views, which lack sufficient
visual cues about surrounding environment. Moreover, natural language contains
complex semantic information thus its correlations with visual inputs are hard
to model merely with cross attention. In this paper, we propose GeoVLN, which
learns Geometry-enhanced visual representation based on slot attention for
robust Visual-and-Language Navigation. The RGB images are compensated with the
corresponding depth maps and normal maps predicted by Omnidata as visual
inputs. Technically, we introduce a two-stage module that combine local slot
attention and CLIP model to produce geometry-enhanced representation from such
input. We employ V&L BERT to learn a cross-modal representation that
incorporate both language and vision informations. Additionally, a novel
multiway attention module is designed, encouraging different phrases of input
instruction to exploit the most related features from visual input. Extensive
experiments demonstrate the effectiveness of our newly designed modules and
show the compelling performance of the proposed method.

GeoVLN 提出了一种几何加强的视觉表示学习方法，基于 Slot Attention 技术和 V&L BERT 模型结合自然语言和多种视觉输入信息实现了强大的视觉和语言导航功能，取得了很好的实验效果。

GeoVLN：使用槽注意力学习几何增强的视觉表征，用于视觉语言导航

GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot  Attention for Vision-and-Language Navigation

Deep Learning has revolutionized our ability to solve complex problems such
as Vision-and-Language Navigation (VLN). This task requires the agent to
navigate to a goal purely based on visual sensory inputs given natural language
instructions. However, prior works formulate the problem as a navigation graph
with a discrete action space. In this work, we lift the agent off the
navigation graph and propose a more complex VLN setting in continuous 3D
reconstructed environments. Our proposed setting, Robo-VLN, more closely mimics
the challenges of real world navigation. Robo-VLN tasks have longer trajectory
lengths, continuous action spaces, and challenges such as obstacles. We provide
a suite of baselines inspired by state-of-the-art works in discrete VLN and
show that they are less effective at this task. We further propose that
decomposing the task into specialized high- and low-level policies can more
effectively tackle this task. With extensive experiments, we show that by using
layered decision making, modularized training, and decoupling reasoning and
imitation, our proposed Hierarchical Cross-Modal (HCM) agent outperforms
existing baselines in all key metrics and sets a new benchmark for Robo-VLN.

该研究论文提出了一种基于连续三维重建环境的机器人视觉和语言导航（Robo-VLN）的设定，并利用分层决策，模块化训练和分离推理和模仿的策略，成功地解决了 Robo-VLN 这一新任务。通过使用 HCM 代理，研究者展示了比现有基线效果更好的结果，为 Robo-VLN 创造了新的基准。