Acquiring a multi-task imitation policy in 3D manipulation poses challenges
in terms of scene understanding and action prediction. Current methods employ
both 3D representation and multi-view 2D representation to predict the poses of
the robot's end-effector. However, they still require a considerable amount of
high-quality robot trajectories, and suffer from limited generalization in
unseen tasks and inefficient execution in long-horizon reasoning. In this
paper, we propose SAM-E, a novel architecture for robot manipulation by
leveraging a vision-foundation model for generalizable scene understanding and
sequence imitation for long-term action reasoning. Specifically, we adopt
Segment Anything (SAM) pre-trained on a huge number of images and promptable
masks as the foundation model for extracting task-relevant features, and employ
parameter-efficient fine-tuning on robot data for a better understanding of
embodied scenarios. To address long-horizon reasoning, we develop a novel
multi-channel heatmap that enables the prediction of the action sequence in a
single pass, notably enhancing execution efficiency. Experimental results from
various instruction-following tasks demonstrate that SAM-E achieves superior
performance with higher execution efficiency compared to the baselines, and
also significantly improves generalization in few-shot adaptation to new tasks.

通过 SAM-E，我们提出了一种新颖的机器人操作体系结构，通过视觉基础模型进行可 generalizable 的场景理解和长期 action 推理的序列模仿，在各种指令跟踪任务中展现出卓越的性能和更高的执行效率，并显著提高了对新任务的少样本适应性。

SAM-E：利用序列模仿与视觉基础模型实现身体操作

SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for  Embodied Manipulation

Neural Radiance Fields (NeRFs) have revolutionized scene novel view
synthesis, offering visually realistic, precise, and robust implicit
reconstructions. While recent approaches enable NeRF editing, such as object
removal, 3D shape modification, or material property manipulation, the manual
annotation prior to such edits makes the process tedious. Additionally,
traditional 2D interaction tools lack an accurate sense of 3D space, preventing
precise manipulation and editing of scenes. In this paper, we introduce a novel
approach, called Blueprint Neural Field (BluNF), to address these editing
issues. BluNF provides a robust and user-friendly 2D blueprint, enabling
intuitive scene editing. By leveraging implicit neural representation, BluNF
constructs a blueprint of a scene using prior semantic and depth information.
The generated blueprint allows effortless editing and manipulation of NeRF
representations. We demonstrate BluNF's editability through an intuitive
click-and-change mechanism, enabling 3D manipulations, such as masking,
appearance modification, and object removal. Our approach significantly
contributes to visual content creation, paving the way for further research in
this area.

通过先前语义和深度信息，BlueNF 构建了场景的蓝图，实现了直观的 3D 编辑和操作，包括遮掩、外观修改和物体去除等，从而为视觉内容创作做出了重要贡献。

蓝色 NF：蓝图神经场

BluNF: Blueprint Neural Field

The manipulation of latent space has recently become an interesting topic in
the field of generative models. Recent research shows that latent directions
can be used to manipulate images towards certain attributes. However,
controlling the generation process of 3D generative models remains a challenge.
In this work, we propose a novel 3D manipulation method that can manipulate
both the shape and texture of the model using text or image-based prompts such
as 'a young face' or 'a surprised face'. We leverage the power of Contrastive
Language-Image Pre-training (CLIP) model and a pre-trained 3D GAN model
designed to generate face avatars, and create a fully differentiable rendering
pipeline to manipulate meshes. More specifically, our method takes an input
latent code and modifies it such that the target attribute specified by a text
or image prompt is present or enhanced, while leaving other attributes largely
unaffected. Our method requires only 5 minutes per manipulation, and we
demonstrate the effectiveness of our approach with extensive results and
comparisons.

该论文介绍了一种基于文本或图像的提示（如 “年轻的脸” 或 “惊讶的脸”）来操纵三维生成模型中形态和纹理的方法，利用了对比语言图像预训练模型（CLIP）和预训练的生成人脸的三维 GAN 模型创建了一个完全可微的渲染管道来操作网格。

文本和图像帮助下的 3D 头像创造与操作

Text and Image Guided 3D Avatar Generation and Manipulation

Object manipulation from 3D visual inputs poses many challenges on building
generalizable perception and policy models. However, 3D assets in existing
benchmarks mostly lack the diversity of 3D shapes that align with real-world
intra-class complexity in topology and geometry. Here we propose SAPIEN
Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over
diverse objects in a full-physics simulator. 3D assets in ManiSkill include
large intra-class topological and geometric variations. Tasks are carefully
chosen to cover distinct types of manipulation challenges. Latest progress in
3D vision also makes us believe that we should customize the benchmark so that
the challenge is inviting to researchers working on 3D deep learning. To this
end, we simulate a moving panoramic camera that returns ego-centric point
clouds or RGB-D images. In addition, we would like ManiSkill to serve a broad
set of researchers interested in manipulation research. Besides supporting the
learning of policies from interactions, we also support
learning-from-demonstrations (LfD) methods, by providing a large number of
high-quality demonstrations (~36,000 successful trajectories, ~1.5M point
cloud/RGB-D frames in total). We provide baselines using 3D deep learning and
LfD algorithms. All code of our benchmark (simulator, environment, SDK, and
baselines) is open-sourced, and a challenge facing interdisciplinary
researchers will be held based on the benchmark.

本文提出了 SAPIEN Manipulation Skill Benchmark（ManiSkill），用于在全物理模拟器中基于多种对象进行操作技能测试，旨在支持互动和演示学习方法，并提供用于 3D 深度学习和 LfD 算法的基线。