In recent years, there has been an explosion of 2D vision models for numerous
tasks such as semantic segmentation, style transfer or scene editing, enabled
by large-scale 2D image datasets. At the same time, there has been renewed
interest in 3D scene representations such as neural radiance fields from
multi-view images. However, the availability of 3D or multiview data is still
substantially limited compared to 2D image datasets, making extending 2D vision
models to 3D data highly desirable but also very challenging. Indeed, extending
a single 2D vision operator like scene editing to 3D typically requires a
highly creative method specialized to that task and often requires per-scene
optimization. In this paper, we ask the question of whether any 2D vision model
can be lifted to make 3D consistent predictions. We answer this question in the
affirmative; our new Lift3D method trains to predict unseen views on feature
spaces generated by a few visual models (i.e. DINO and CLIP), but then
generalizes to novel vision operators and tasks, such as style transfer,
super-resolution, open vocabulary segmentation and image colorization; for some
of these tasks, there is no comparable previous 3D method. In many cases, we
even outperform state-of-the-art methods specialized for the task in question.
Moreover, Lift3D is a zero-shot method, in the sense that it requires no
task-specific training, nor scene-specific optimization.

将 2D 视觉模型扩展到 3D 数据是一个具有挑战性的任务，本文提出了 Lift3D 方法，通过训练预测基于 DINO 和 CLIP 的特征空间的未见视图，实现了在风格转换、超分辨率、开放词汇分割和图像上色等任务上的泛化性能优于专门针对这些任务的最先进方法。

Lift3D：将任何 2D 视觉模型升级为 3D 的零样本方法

Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D

We marry diffusion policies and 3D scene representations for robot
manipulation. Diffusion policies learn the action distribution conditioned on
the robot and environment state using conditional diffusion models. They have
recently shown to outperform both deterministic and alternative
state-conditioned action distribution learning methods. 3D robot policies use
3D scene feature representations aggregated from a single or multiple camera
views using sensed depth. They have shown to generalize better than their 2D
counterparts across camera viewpoints. We unify these two lines of work and
present 3D Diffuser Actor, a neural policy architecture that, given a language
instruction, builds a 3D representation of the visual scene and conditions on
it to iteratively denoise 3D rotations and translations for the robot's
end-effector. At each denoising iteration, our model represents end-effector
pose estimates as 3D scene tokens and predicts the 3D translation and rotation
error for each of them, by featurizing them using 3D relative attention to
other 3D visual and language tokens. 3D Diffuser Actor sets a new
state-of-the-art on RLBench with an absolute performance gain of 16.3% over the
current SOTA on a multi-view setup and an absolute gain of 13.1% on a
single-view setup. On the CALVIN benchmark, it outperforms the current SOTA in
the setting of zero-shot unseen scene generalization by being able to
successfully run 0.2 more tasks, a 7% relative increase. It also works in the
real world from a handful of demonstrations. We ablate our model's
architectural design choices, such as 3D scene featurization and 3D relative
attentions, and show they all help generalization. Our results suggest that 3D
scene representations and powerful generative modeling are keys to efficient
robot learning from demonstrations.

我们结合扩散策略和 3D 场景表示来实现机器人操纵，提出了一种名为 3D Diffuser Actor 的神经策略体系结构，它在给定语言指令的情况下构建视觉场景的 3D 表示，并在此基础上迭代性地对机器人的末端执行器进行 3D 旋转和平移噪声去除。通过实验验证，我们的模型在机器人学习领域取得了显著的性能提升。

3D 扩散行为者：带有 3D 场景表示的策略扩散

3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

3D scene representations have gained immense popularity in recent years.
Methods that use Neural Radiance fields are versatile for traditional tasks
such as novel view synthesis. In recent times, some work has emerged that aims
to extend the functionality of NeRF beyond view synthesis, for semantically
aware tasks such as editing and segmentation using 3D feature field
distillation from 2D foundation models. However, these methods have two major
limitations: (a) they are limited by the rendering speed of NeRF pipelines, and
(b) implicitly represented feature fields suffer from continuity artifacts
reducing feature quality. Recently, 3D Gaussian Splatting has shown
state-of-the-art performance on real-time radiance field rendering. In this
work, we go one step further: in addition to radiance field rendering, we
enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D
foundation model distillation. This translation is not straightforward: naively
incorporating feature fields in the 3DGS framework leads to warp-level
divergence. We propose architectural and training changes to efficiently avert
this problem. Our proposed method is general, and our experiments showcase
novel view semantic segmentation, language-guided editing and segment anything
through learning feature fields from state-of-the-art 2D foundation models such
as SAM and CLIP-LSeg. Across experiments, our distillation method is able to
provide comparable or better results, while being significantly faster to both
train and render. Additionally, to the best of our knowledge, we are the first
method to enable point and bounding-box prompting for radiance field
manipulation, by leveraging the SAM model. Project website at:
this https URL

通过 2D 基础模型的 3D 特征场蒸馏，我们提出了一种通用的方法，用于实现新颖视图的语义分割、语言引导编辑和任意维度语义特征的 3D 高斯喷洒，提供可比较或更好的结果，并且训练和渲染速度显著更快。