It is a long-standing problem in robotics to develop agents capable of
executing diverse manipulation tasks from visual observations in unstructured
real-world environments. To achieve this goal, the robot needs to have a
comprehensive understanding of the 3D structure and semantics of the scene. In
this work, we present $\textbf{GNFactor}$, a visual behavior cloning agent for
multi-task robotic manipulation with $\textbf{G}$eneralizable $\textbf{N}$eural
feature $\textbf{F}$ields. GNFactor jointly optimizes a generalizable neural
field (GNF) as a reconstruction module and a Perceiver Transformer as a
decision-making module, leveraging a shared deep 3D voxel representation. To
incorporate semantics in 3D, the reconstruction module utilizes a
vision-language foundation model ($\textit{e.g.}$, Stable Diffusion) to distill
rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3
real robot tasks and perform detailed ablations on 10 RLBench tasks with a
limited number of demonstrations. We observe a substantial improvement of
GNFactor over current state-of-the-art methods in seen and unseen tasks,
demonstrating the strong generalization ability of GNFactor. Our project
website is this https URL .

用视觉观察来实现机器人在无结构的真实环境中执行多样化操作任务一直是机器人学领域的一个长期存在的问题。本文提出了 GNFactor，它是一个具有通用化神经特征场的多任务机器人操作的可视化行为复制代理系统，通过共享的深度三维体素表示，GNFactor 同时优化通用化神经场作为重建模块和 Perceiver Transformer 作为决策模块，以实现对 3D 场景的全面理解。

GNFactor：通用神经特征场下的多任务真实机器人学习

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural  Feature Fields

From an image of a person, we can easily infer the natural 3D pose and shape
of the person even if ambiguity exists. This is because we have a mental model
that allows us to imagine a person's appearance at different viewing directions
from a given image and utilize the consistency between them for inference.
However, existing human mesh recovery methods only consider the direction in
which the image was taken due to their structural limitations. Hence, we
propose "Implicit 3D Human Mesh Recovery (ImpHMR)" that can implicitly imagine
a person in 3D space at the feature-level via Neural Feature Fields. In ImpHMR,
feature fields are generated by CNN-based image encoder for a given image.
Then, the 2D feature map is volume-rendered from the feature field for a given
viewing direction, and the pose and shape parameters are regressed from the
feature. To utilize consistency with pose and shape from unseen-view, if there
are 3D labels, the model predicts results including the silhouette from an
arbitrary direction and makes it equal to the rotated ground-truth. In the case
of only 2D labels, we perform self-supervised learning through the constraint
that the pose and shape parameters inferred from different directions should be
the same. Extensive evaluations show the efficacy of the proposed method.

通过神经特征场可以隐含模拟出人类在 3D 空间中的姿态和形状，从而提出了隐式三维人体网格恢复 (ImpHMR) 方法，并通过 Consistency and Self-supervised 的学习方式实现了姿态和形状参数的预测。