An open problem in mobile manipulation is how to represent objects and scenes
in a unified manner, so that robots can use it both for navigating in the
environment and manipulating objects. The latter requires capturing intricate
geometry while understanding fine-grained semantics, whereas the former
involves capturing the complexity inherit to an expansive physical scale. In
this work, we present GeFF (Generalizable Feature Fields), a scene-level
generalizable neural feature field that acts as a unified representation for
both navigation and manipulation that performs in real-time. To do so, we treat
generative novel view synthesis as a pre-training task, and then align the
resulting rich scene priors with natural language via CLIP feature
distillation. We demonstrate the effectiveness of this approach by deploying
GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's
ability to generalize to open-set objects as well as running time, when
performing open-vocabulary mobile manipulation in dynamic scenes.

本研究提出了一种名为 GeFF（通用特征场）的场景级通用神经特征场，作为导航和操作的统一表示，实时性能良好，并在四足机器人上部署，评估了其对开放集对象和动态场景中进行移动操作的泛化能力和运行时间。

移动机器人操作的学习通用特征场

Learning Generalizable Feature Fields for Mobile Manipulation

Image captioning models tend to describe images in an object-centric way,
emphasising visible objects. But image descriptions can also abstract away from
objects and describe the type of scene depicted. In this paper, we explore the
potential of a state-of-the-art Vision and Language model, VinVL, to caption
images at the scene level using (1) a novel dataset which pairs images with
both object-centric and scene descriptions. Through (2) an in-depth analysis of
the effect of the fine-tuning, we show (3) that a small amount of curated data
suffices to generate scene descriptions without losing the capability to
identify object-level concepts in the scene; the model acquires a more holistic
view of the image compared to when object-centric descriptions are generated.
We discuss the parallels between these results and insights from computational
and cognitive science research on scene perception.

使用 Vision and Language 模型，结合场景和物体描述的数据集，本文分析了其微调的效果，揭示了小量精选数据即可生成不失物体级别描述的场景描述，并讨论与计算和认知科学研究的并行之处。