In recent years, there has been an explosion of 2D vision models for numerous
tasks such as semantic segmentation, style transfer or scene editing, enabled
by large-scale 2D image datasets. At the same time, there has been renewed
interest in 3D scene representations such as neural radiance fields from
multi-view images. However, the availability of 3D or multiview data is still
substantially limited compared to 2D image datasets, making extending 2D vision
models to 3D data highly desirable but also very challenging. Indeed, extending
a single 2D vision operator like scene editing to 3D typically requires a
highly creative method specialized to that task and often requires per-scene
optimization. In this paper, we ask the question of whether any 2D vision model
can be lifted to make 3D consistent predictions. We answer this question in the
affirmative; our new Lift3D method trains to predict unseen views on feature
spaces generated by a few visual models (i.e. DINO and CLIP), but then
generalizes to novel vision operators and tasks, such as style transfer,
super-resolution, open vocabulary segmentation and image colorization; for some
of these tasks, there is no comparable previous 3D method. In many cases, we
even outperform state-of-the-art methods specialized for the task in question.
Moreover, Lift3D is a zero-shot method, in the sense that it requires no
task-specific training, nor scene-specific optimization.

将 2D 视觉模型扩展到 3D 数据是一个具有挑战性的任务，本文提出了 Lift3D 方法，通过训练预测基于 DINO 和 CLIP 的特征空间的未见视图，实现了在风格转换、超分辨率、开放词汇分割和图像上色等任务上的泛化性能优于专门针对这些任务的最先进方法。

Lift3D：将任何 2D 视觉模型升级为 3D 的零样本方法

Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D

Natural language processing and 2D vision models have attained remarkable
proficiency on many tasks primarily by escalating the scale of training data.
However, 3D vision tasks have not seen the same progress, in part due to the
challenges of acquiring high-quality 3D data. In this work, we present
Objaverse-XL, a dataset of over 10 million 3D objects. Our dataset comprises
deduplicated 3D objects from a diverse set of sources, including manually
designed objects, photogrammetry scans of landmarks and everyday items, and
professional scans of historic and antique artifacts. Representing the largest
scale and diversity in the realm of 3D datasets, Objaverse-XL enables
significant new possibilities for 3D vision. Our experiments demonstrate the
improvements enabled with the scale provided by Objaverse-XL. We show that by
training Zero123 on novel view synthesis, utilizing over 100 million multi-view
rendered images, we achieve strong zero-shot generalization abilities. We hope
that releasing Objaverse-XL will enable further innovations in the field of 3D
vision at scale.

本研究提出了一个新的数据集 Objaverse-XL，该数据集包含超过 1000 万个多样化来源的 3D 物体，从手工设计物体到历史文物数字扫描等，旨在帮助 3D 视觉领域得到更大的发展以及在许多方面取得成果