Vision-language navigation (VLN) requires an agent to navigate through an 3D
environment based on visual observations and natural language instructions. It
is clear that the pivotal factor for successful navigation lies in the
comprehensive scene understanding. Previous VLN agents employ monocular
frameworks to extract 2D features of perspective views directly. Though
straightforward, they struggle for capturing 3D geometry and semantics, leading
to a partial and incomplete environment representation. To achieve a
comprehensive 3D representation with fine-grained details, we introduce a
Volumetric Environment Representation (VER), which voxelizes the physical world
into structured 3D cells. For each cell, VER aggregates multi-view 2D features
into such a unified 3D space via 2D-3D sampling. Through coarse-to-fine feature
extraction and multi-task learning for VER, our agent predicts 3D occupancy, 3D
room layout, and 3D bounding boxes jointly. Based on online collected VERs, our
agent performs volume state estimation and builds episodic memory for
predicting the next step. Experimental results show our environment
representations from multi-task learning lead to evident performance gains on
VLN. Our model achieves state-of-the-art performance across VLN benchmarks
(R2R, REVERIE, and R4R).

基于视觉观察和自然语言指令，本论文提出了一种基于体素化环境表示的视觉语言导航模型，在多任务学习的影响下，预测 3D 占用、3D 房间布局和 3D 边界框，并通过在线采集的环境表示进行体积状态估计和建立序列记忆，取得了在 VLN 基准测试（R2R，REVERIE 和 R4R）上最先进的性能。