Trained with an unprecedented scale of data, large language models (LLMs)
like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities
from model scaling. Such a trend underscored the potential of training LLMs
with unlimited language data, advancing the development of a universal embodied
agent. In this work, we introduce the NavGPT, a purely LLM-based
instruction-following navigation agent, to reveal the reasoning capability of
GPT models in complex embodied scenes by performing zero-shot sequential action
prediction for vision-and-language navigation (VLN). At each step, NavGPT takes
the textual descriptions of visual observations, navigation history, and future
explorable directions as inputs to reason the agent's current status, and makes
the decision to approach the target. Through comprehensive experiments, we
demonstrate NavGPT can explicitly perform high-level planning for navigation,
including decomposing instruction into sub-goal, integrating commonsense
knowledge relevant to navigation task resolution, identifying landmarks from
observed scenes, tracking navigation progress, and adapting to exceptions with
plan adjustment. Furthermore, we show that LLMs is capable of generating
high-quality navigational instructions from observations and actions along a
path, as well as drawing accurate top-down metric trajectory given the agent's
navigation history. Despite the performance of using NavGPT to zero-shot R2R
tasks still falling short of trained models, we suggest adapting multi-modality
inputs for LLMs to use as visual navigation agents and applying the explicit
reasoning of LLMs to benefit learning-based models.

本篇论文中，我们使用大语言模型（LLMs）为基础，介绍了一款纯 LLMs 导航代理 NavGPT，通过对视觉和语言导航（VLN）进行零样本顺序动作预测，揭示了 GPT 模型在复杂体验场景中的推理能力，以及它在导航中的应用，通过全面的实验，展示了 NavGPT 可以分解指令成子目标、对与导航任务相关的常识知识进行集成以及适应特殊情况等高级规划能力。