We present a two-stage text-to-3D generation system, namely 3DTopia, which
generates high-quality general 3D assets within 5 minutes using hybrid
diffusion priors. The first stage samples from a 3D diffusion prior directly
learned from 3D data. Specifically, it is powered by a text-conditioned
tri-plane latent diffusion model, which quickly generates coarse 3D samples for
fast prototyping. The second stage utilizes 2D diffusion priors to further
refine the texture of coarse 3D models from the first stage. The refinement
consists of both latent and pixel space optimization for high-quality texture
generation. To facilitate the training of the proposed system, we clean and
caption the largest open-source 3D dataset, Objaverse, by combining the power
of vision language models and large language models. Experiment results are
reported qualitatively and quantitatively to show the performance of the
proposed system. Our codes and models are available at
this https URL

我们提出了一个两阶段的文本到 3D 生成系统，名为 3DTopia，它使用混合扩散先验在 5 分钟内生成高质量的通用 3D 资产。第一阶段直接从 3D 数据学习的 3D 扩散先验中采样，由文本条件的三视图潜在扩散模型提供动力，快速生成用于快速原型设计的粗糙 3D 样本。第二阶段利用 2D 扩散先验进一步优化第一阶段中粗糙 3D 模型的纹理。优化包括潜在空间和像素空间的优化，用于生成高质量的纹理。为了促进所提出系统的训练，我们通过结合视觉语言模型和大型语言模型清洗和加标最大的开源 3D 数据集 Objaverse。实验证据定性和定量地展示了所提出系统的性能。我们的代码和模型可在此 https 网址获得。

3DTopia: 带混合扩散先验的大规模文本到 3D 生成模型

3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors

The field of generative AI has a transformative impact on various areas,
including virtual reality, autonomous driving, the metaverse, gaming, and
robotics. Among these applications, 3D object generation techniques are of
utmost importance. This technique has unlocked fresh avenues in the realm of
creating, customizing, and exploring 3D objects. However, the quality and
diversity of existing 3D object generation methods are constrained by the
inadequacies of existing 3D object datasets, including issues related to text
quality, the incompleteness of multi-modal data representation encompassing 2D
rendered images and 3D assets, as well as the size of the dataset. In order to
resolve these issues, we present UniG3D, a unified 3D object generation dataset
constructed by employing a universal data transformation pipeline on Objaverse
and ShapeNet datasets. This pipeline converts each raw 3D model into
comprehensive multi-modal data representation <text, image, point cloud, mesh>
by employing rendering engines and multi-modal models. These modules ensure the
richness of textual information and the comprehensiveness of data
representation. Remarkably, the universality of our pipeline refers to its
ability to be applied to any 3D dataset, as it only requires raw 3D data. The
selection of data sources for our dataset is based on their scale and quality.
Subsequently, we assess the effectiveness of our dataset by employing Point-E
and SDFusion, two widely recognized methods for object generation, tailored to
the prevalent 3D representations of point clouds and signed distance functions.
Our dataset is available at: this https URL

UniG3D 是一个统一的 3D 对象生成数据集，采用 Objaverse 和 ShapeNet 数据集的通用数据转换管道，并使用渲染引擎和多模态模型将每个原始的 3D 模型转换成包含文本、图像、点云和网格的综合多模态数据表示，该数据集可应用于任何 3D 数据集中。