We present an optimization study of the Vision-Language Frontier Maps (VLFM)
applied to the Object Goal Navigation task in robotics. Our work evaluates the
efficiency and performance of various vision-language models, object detectors,
segmentation models, and multi-modal comprehension and Visual Question
Answering modules. Using the $\textit{val-mini}$ and $\textit{val}$ splits of
Habitat-Matterport 3D dataset, we conduct experiments on a desktop with limited
VRAM. We propose a solution that achieves a higher success rate (+1.55%)
improving over the VLFM BLIP-2 baseline without substantial success-weighted
path length loss while requiring $\textbf{2.3 times}$ less video memory. Our
findings provide insights into balancing model performance and computational
efficiency, suggesting effective deployment strategies for resource-limited
environments.

我们对机器人的目标导航任务中应用的视觉语言前沿地图（VLFM）进行了优化研究。我们评估了各种视觉语言模型、目标检测器、分割模型以及多模态理解和视觉问答模块的效率和性能。在有限的 VRAM 桌面上，我们提出了一个解决方案，它在不丢失重要路径长度损失的情况下，比 VLFM BLIP-2 基线取得了更高的成功率（+1.55%），并且需要的视频内存减少了 2.3 倍。我们的研究结果揭示了在资源有限的环境中平衡模型性能和计算效率的洞见，并提出了有效的部署策略。

在零样本机器人导航中平衡性能和效率

Balancing Performance and Efficiency in Zero-shot Robotic Navigation

Understanding how humans leverage semantic knowledge to navigate unfamiliar
environments and decide where to explore next is pivotal for developing robots
capable of human-like search behaviors. We introduce a zero-shot navigation
approach, Vision-Language Frontier Maps (VLFM), which is inspired by human
reasoning and designed to navigate towards unseen semantic objects in novel
environments. VLFM builds occupancy maps from depth observations to identify
frontiers, and leverages RGB observations and a pre-trained vision-language
model to generate a language-grounded value map. VLFM then uses this map to
identify the most promising frontier to explore for finding an instance of a
given target object category. We evaluate VLFM in photo-realistic environments
from the Gibson, Habitat-Matterport 3D (HM3D), and Matterport 3D (MP3D)
datasets within the Habitat simulator. Remarkably, VLFM achieves
state-of-the-art results on all three datasets as measured by success weighted
by path length (SPL) for the Object Goal Navigation task. Furthermore, we show
that VLFM's zero-shot nature enables it to be readily deployed on real-world
robots such as the Boston Dynamics Spot mobile manipulation platform. We deploy
VLFM on Spot and demonstrate its capability to efficiently navigate to target
objects within an office building in the real world, without any prior
knowledge of the environment. The accomplishments of VLFM underscore the
promising potential of vision-language models in advancing the field of
semantic navigation. Videos of real-world deployment can be viewed at
naoki.io/vlfm.

人类如何利用语义知识在陌生环境中导航并决定下一步探索的方式对于开发能够展现类似人类搜索行为的机器人来说至关重要。我们介绍了一种零样本导航方法，即视觉语言前沿地图（VLFM），其受人类推理的启发，并设计用于在新环境中驶向未见过语义对象。VLFM 根据深度观测建立占用地图，以识别前沿，并利用 RGB 观测和预训练的视觉语言模型生成基于语言的价值地图。然后，VLFM 使用该地图来识别探索最有希望的前沿，以寻找给定目标对象类别的实例。我们在 Gibson、Habitat-Matterport 3D（HM3D）和 Matterport 3D（MP3D）数据集的逼真环境中，在 Habitat 模拟器中评估了 VLFM。令人惊讶的是，VLFM 在对象目标导航任务的成功路径长度（SPL）方面，在所有三个数据集上都实现了最先进的结果。此外，我们还展示了 VLFM 的零样本特性使其能够轻松部署在现实世界的机器人上，如波士顿动力公司的 Spot 移动操作平台。我们在 Spot 上部署了 VLFM，并展示了它在现实世界的办公楼中高效导航到目标对象的能力，而无需任何对环境的先验知识。VLFM 的成就凸显了视觉语言模型在推进语义导航领域的巨大潜力。可以在 naoki.io/vlfm 上观看实际部署的视频。