The vision-and-language navigation (VLN) task necessitates an agent to
perceive the surroundings, follow natural language instructions, and act in
photo-realistic unseen environments. Most of the existing methods employ the
entire image or object features to represent navigable viewpoints. However,
these representations are insufficient for proper action prediction, especially
for the REVERIE task, which uses concise high-level instructions, such as
''Bring me the blue cushion in the master bedroom''. To address enhancing
representation, we propose an augmented commonsense knowledge model (ACK) to
leverage commonsense information as a spatio-temporal knowledge graph for
improving agent navigation. Specifically, the proposed approach involves
constructing a knowledge base by retrieving commonsense information from
ConceptNet, followed by a refinement module to remove noisy and irrelevant
knowledge. We further present ACK which consists of knowledge graph-aware
cross-modal and concept aggregation modules to enhance visual representation
and visual-textual data alignment by integrating visible objects, commonsense
knowledge, and concept history, which includes object and knowledge temporal
information. Moreover, we add a new pipeline for the commonsense-based
decision-making process which leads to more accurate local action prediction.
Experimental results demonstrate our proposed model noticeably outperforms the
baseline and archives the state-of-the-art on the REVERIE benchmark.

通过引入增强型常识知识模型 (ACK)，利用常识信息的时空知识图，增强代理导航，通过知识图感知跨模态和概念聚合模块来提升可视化表示和可视化文本数据对齐，同时引入基于常识的决策过程，实现更准确的本地动作预测。