An open problem in mobile manipulation is how to represent objects and scenes
in a unified manner, so that robots can use it both for navigating in the
environment and manipulating objects. The latter requires capturing intricate
geometry while understanding fine-grained semantics, whereas the former
involves capturing the complexity inherit to an expansive physical scale. In
this work, we present GeFF (Generalizable Feature Fields), a scene-level
generalizable neural feature field that acts as a unified representation for
both navigation and manipulation that performs in real-time. To do so, we treat
generative novel view synthesis as a pre-training task, and then align the
resulting rich scene priors with natural language via CLIP feature
distillation. We demonstrate the effectiveness of this approach by deploying
GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's
ability to generalize to open-set objects as well as running time, when
performing open-vocabulary mobile manipulation in dynamic scenes.

本研究提出了一种名为 GeFF（通用特征场）的场景级通用神经特征场，作为导航和操作的统一表示，实时性能良好，并在四足机器人上部署，评估了其对开放集对象和动态场景中进行移动操作的泛化能力和运行时间。

移动机器人操作的学习通用特征场

Learning Generalizable Feature Fields for Mobile Manipulation

Object navigation (ObjectNav) requires an agent to navigate through unseen
environments to find queried objects. Many previous methods attempted to solve
this task by relying on supervised or reinforcement learning, where they are
trained on limited household datasets with close-set objects. However, two key
challenges are unsolved: understanding free-form natural language instructions
that demand open-set objects, and generalizing to new environments in a
zero-shot manner. Aiming to solve the two challenges, in this paper, we propose
OpenFMNav, an Open-set Foundation Model based framework for zero-shot object
Navigation. We first unleash the reasoning abilities of large language models
(LLMs) to extract proposed objects from natural language instructions that meet
the user's demand. We then leverage the generalizability of large vision
language models (VLMs) to actively discover and detect candidate objects from
the scene, building a Versatile Semantic Score Map (VSSM). Then, by conducting
common sense reasoning on VSSM, our method can perform effective
language-guided exploration and exploitation of the scene and finally reach the
goal. By leveraging the reasoning and generalizing abilities of foundation
models, our method can understand free-form human instructions and perform
effective open-set zero-shot navigation in diverse environments. Extensive
experiments on the HM3D ObjectNav benchmark show that our method surpasses all
the strong baselines on all metrics, proving our method's effectiveness.
Furthermore, we perform real robot demonstrations to validate our method's
open-set-ness and generalizability to real-world environments.

我们提出了 OpenFMNav，一种基于开放集对象导航的零样本模型，通过利用大型语言模型和视觉语言模型的推理和泛化能力，能够理解自由形式的人类指令，并在各种环境中进行有效的零样本导航，通过对 HM3D ObjectNav 基准测试的广泛实验，证明了我们方法的有效性，并通过真实机器人演示验证了这种方法对真实环境的开放性和泛化能力。