Recent works have shown that visual pretraining on egocentric datasets using
masked autoencoders (MAE) can improve generalization for downstream robotics
tasks. However, these approaches pretrain only on 2D images, while many
robotics applications require 3D scene understanding. In this work, we propose
3D-MVP, a novel approach for 3D multi-view pretraining using masked
autoencoders. We leverage Robotic View Transformer (RVT), which uses a
multi-view transformer to understand the 3D scene and predict gripper pose
actions. We split RVT's multi-view transformer into visual encoder and action
decoder, and pretrain its visual encoder using masked autoencoding on
large-scale 3D datasets such as Objaverse. We evaluate 3D-MVP on a suite of
virtual robot manipulation tasks and demonstrate improved performance over
baselines. We also show promising results on a real robot platform with minimal
finetuning. Our results suggest that 3D-aware pretraining is a promising
approach to improve sample efficiency and generalization of vision-based
robotic manipulation policies. We will release code and pretrained models for
3D-MVP to facilitate future research. Project site:
this https URL

提出了 3D-MVP，一种使用遮罩自编码器进行 3D 多视图预训练的新方法，通过在大规模 3D 数据集上使用遮罩自编码器对其视觉编码器进行预训练，评估了其在虚拟机器人操作任务中的性能并展示了超过基线的改进，同时在实际机器人平台上也取得了令人期待的结果。

3D-MVP：用于机器人操纵的三维多视角预训练

3D-MVP: 3D Multiview Pretraining for Robotic Manipulation

Generating instructional images of human daily actions from an egocentric
viewpoint serves a key step towards efficient skill transfer. In this paper, we
introduce a novel problem -- egocentric action frame generation. The goal is to
synthesize the action frame conditioning on the user prompt question and an
input egocentric image that captures user's environment. Notably, existing
egocentric datasets lack the detailed annotations that describe the execution
of actions. Additionally, the diffusion-based image manipulation models fail to
control the state change of an action within the corresponding egocentric image
pixel space. To this end, we finetune a visual large language model (VLLM) via
visual instruction tuning for curating the enriched action descriptions to
address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO)
action frame generation using image and text embeddings from VLLM as additional
conditioning. We validate our proposed model on two egocentric datasets --
Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior
image manipulation models in both quantitative and qualitative evaluation. We
also conduct detailed ablation studies and analysis to provide insights on our
method.

从以自身为中心的视角生成人类日常行为的指导性图像是有效的技能传递的一个关键步骤。本文提出了一个新颖的问题 —— 以自身为中心的动作帧生成。目标是根据用户提示问题和捕捉用户环境的输入自身中心图像，合成动作帧。值得注意的是，现有的自身中心数据集缺乏描述动作执行的详细注释。此外，基于扩散的图像操作模型无法控制动作在相应自身中心图像像素空间内的状态变化。为此，我们通过视觉指导优化自然语言大型模型（VLLM），以充实的动作描述进行微调，以解决我们提出的问题。此外，我们还提出利用来自 VLLM 的图像和文本嵌入作为附加条件的 Learn EGOcentric（LEGO）动作帧生成方法。我们在两个自身中心数据集 ——Ego4D 和 Epic-Kitchens 上验证了我们的模型。我们的实验证明，在定量和定性评估方面，我们的提出的模型相较于先前的图像操作模型有显著改进。我们还进行了详细的消融研究和分析，为我们的方法提供了深入认识。

LEGO: 通过视觉指导调整学习自我中心动作帧生成

LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction  Tuning

In this paper, we study a problem of egocentric scene understanding, i.e.,
predicting depths and surface normals from an egocentric image. Egocentric
scene understanding poses unprecedented challenges: (1) due to large head
movements, the images are taken from non-canonical viewpoints (i.e., tilted
images) where existing models of geometry prediction do not apply; (2) dynamic
foreground objects including hands constitute a large proportion of visual
scenes. These challenges limit the performance of the existing models learned
from large indoor datasets, such as ScanNet and NYUv2, which comprise
predominantly upright images of static scenes. We present a multimodal spatial
rectifier that stabilizes the egocentric images to a set of reference
directions, which allows learning a coherent visual representation. Unlike
unimodal spatial rectifier that often produces excessive perspective warp for
egocentric images, the multimodal spatial rectifier learns from multiple
directions that can minimize the impact of the perspective warp. To learn
visual representations of the dynamic foreground objects, we present a new
dataset called EDINA (Egocentric Depth on everyday INdoor Activities) that
comprises more than 500K synchronized RGBD frames and gravity directions.
Equipped with the multimodal spatial rectifier and the EDINA dataset, our
proposed method on single-view depth and surface normal estimation
significantly outperforms the baselines not only on our EDINA dataset, but also
on other popular egocentric datasets, such as First Person Hand Action (FPHA)
and EPIC-KITCHENS.

本文提出了一种多模态空间矫正器解决自我中心图像中景深和表面法线预测的挑战，同时提出了一个新的数据集 EDINA 并使用多模态空间矫正器进行单视角深度和表面法线预测，在常见的自我中心图像数据集上优于基准模型。