Recent works on text-to-3d generation show that using only 2D diffusion
supervision for 3D generation tends to produce results with inconsistent
appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals
with extra legs). Existing methods mainly address this issue by retraining
diffusion models with images rendered from 3D data to ensure multi-view
consistency while struggling to balance 2D generation quality with 3D
consistency. In this paper, we present a new framework Sculpt3D that equips the
current pipeline with explicit injection of 3D priors from retrieved reference
objects without re-training the 2D diffusion model. Specifically, we
demonstrate that high-quality and diverse 3D geometry can be guaranteed by
keypoints supervision through a sparse ray sampling approach. Moreover, to
ensure accurate appearances of different views, we further modulate the output
of the 2D diffusion model to the correct patterns of the template views without
altering the generated object's style. These two decoupled designs effectively
harness 3D information from reference objects to generate 3D objects while
preserving the generation quality of the 2D diffusion model. Extensive
experiments show our method can largely improve the multi-view consistency
while retaining fidelity and diversity. Our project page is available at:
this https URL

使用稀疏射线采样方法通过关键点监督，实现了在当前管道中显式注入来自检索参考对象的 3D 先验，以确保高质量和多样化的 3D 几何，同时保持 2D 扩散模型的生成质量和多视角一致性。

Sculpt3D：多视角一致的基于稀疏三维先验的文本生成 3D

Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D  Prior

Text-to-3D generation by distilling pretrained large-scale text-to-image
diffusion models has shown great promise but still suffers from inconsistent 3D
geometric structures (Janus problems) and severe artifacts. The aforementioned
problems mainly stem from 2D diffusion models lacking 3D awareness during the
lifting. In this work, we present GeoDream, a novel method that incorporates
explicit generalized 3D priors with 2D diffusion priors to enhance the
capability of obtaining unambiguous 3D consistent geometric structures without
sacrificing diversity or fidelity. Specifically, we first utilize a multi-view
diffusion model to generate posed images and then construct cost volume from
the predicted image, which serves as native 3D geometric priors, ensuring
spatial consistency in 3D space. Subsequently, we further propose to harness 3D
geometric priors to unlock the great potential of 3D awareness in 2D diffusion
priors via a disentangled design. Notably, disentangling 2D and 3D priors
allows us to refine 3D geometric priors further. We justify that the refined 3D
geometric priors aid in the 3D-aware capability of 2D diffusion priors, which
in turn provides superior guidance for the refinement of 3D geometric priors.
Our numerical and visual comparisons demonstrate that GeoDream generates more
3D consistent textured meshes with high-resolution realistic renderings (i.e.,
1024 $\times$ 1024) and adheres more closely to semantic coherence.

通过结合三维几何先验和二维扩散先验，GeoDream 方法能够生成具有一致的三维几何结构的文本到三维模型，并提供更高分辨率、真实感的渲染和更好的语义一致性。

GeoDream: 高保真和一致的 3D 生成中的解耦 2D 和几何先验

GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and  Consistent 3D Generation

We introduce GROOT, an imitation learning method for learning robust policies
with object-centric and 3D priors. GROOT builds policies that generalize beyond
their initial training conditions for vision-based manipulation. It constructs
object-centric 3D representations that are robust toward background changes and
camera views and reason over these representations using a transformer-based
policy. Furthermore, we introduce a segmentation correspondence model that
allows policies to generalize to new objects at test time. Through
comprehensive experiments, we validate the robustness of GROOT policies against
perceptual variations in simulated and real-world environments. GROOT's
performance excels in generalization over background changes, camera viewpoint
shifts, and the presence of new object instances, whereas both state-of-the-art
end-to-end learning methods and object proposal-based approaches fall short. We
also extensively evaluate GROOT policies on real robots, where we demonstrate
the efficacy under very wild changes in setup. More videos and model details
can be found in the appendix and the project website:
this https URL .

GROOT 是一种基于模仿学习的方法，利用物体中心化和 3D 先验知识来学习鲁棒策略，通过构建鲁棒的物体中心化 3D 表示，并使用基于 Transformer 的策略对这些表示进行推理。此外，我们还引入了分割对应模型，使策略能够在测试时适用于新的物体。通过全面的实验，我们验证了 GROOT 在模拟和真实环境中对感知变化的鲁棒性，GROOT 在应对背景变化、摄像机视角变化和新物体实例等方面具有优越的泛化能力，而最先进的端到端学习方法和基于物体提议的方法都表现不佳。我们还在真实机器人上广泛评估了 GROOT 策略，在非常复杂的环境变化下展示了其有效性。

学习具有物体为中心的三维表示的可泛化的操作策略

Learning Generalizable Manipulation Policies with Object-Centric 3D  Representations

Current popular backbones in computer vision, such as Vision Transformers
(ViT) and ResNets are trained to perceive the world from 2D images. However, to
more effectively understand 3D structural priors in 2D backbones, we propose
Mask3D to leverage existing large-scale RGB-D data in a self-supervised
pre-training to embed these 3D priors into 2D learned feature representations.
In contrast to traditional 3D contrastive learning paradigms requiring 3D
reconstructions or multi-view correspondences, our approach is simple: we
formulate a pre-text reconstruction task by masking RGB and depth patches in
individual RGB-D frames. We demonstrate the Mask3D is particularly effective in
embedding 3D priors into the powerful 2D ViT backbone, enabling improved
representation learning for various scene understanding tasks, such as semantic
segmentation, instance segmentation and object detection. Experiments show that
Mask3D notably outperforms existing self-supervised 3D pre-training approaches
on ScanNet, NYUv2, and Cityscapes image understanding tasks, with an
improvement of +6.5% mIoU against the state-of-the-art Pri3D on ScanNet image
semantic segmentation.

提出了一种名为 Mask3D 的预训练方法，可以将现有的大规模 RGB-D 数据应用于自监督预训练中，将 3D 先验嵌入到 2D 的学习特征中，并对多个场景理解任务产生了改进，尤其是语义分割。