While text-to-3D and image-to-3D generation tasks have received considerable
attention, one important but under-explored field between them is controllable
text-to-3D generation, which we mainly focus on in this work. To address this
task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network
architecture designed to enhance existing pre-trained multi-view diffusion
models by integrating additional input conditions, such as edge, depth, normal,
and scribble maps. Our innovation lies in the introduction of a conditioning
module that controls the base diffusion model using both local and global
embeddings, which are computed from the input condition images and camera
poses. Once trained, MVControl is able to offer 3D diffusion guidance for
optimization-based 3D generation. And, 2) we propose an efficient multi-stage
3D generation pipeline that leverages the benefits of recent large
reconstruction models and score distillation algorithm. Building upon our
MVControl architecture, we employ a unique hybrid diffusion guidance method to
direct the optimization process. In pursuit of efficiency, we adopt 3D
Gaussians as our representation instead of the commonly used implicit
representations. We also pioneer the use of SuGaR, a hybrid representation that
binds Gaussians to mesh triangle faces. This approach alleviates the issue of
poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained
geometry on the mesh. Extensive experiments demonstrate that our method
achieves robust generalization and enables the controllable generation of
high-quality 3D content.

本研究主要关注在控制性文本生成 3D 场景方面的多视图控制网络架构，该网络结合额外的输入条件，如边缘、深度、法线和素描图，使用局部和全局嵌入来控制基础扩散模型，并通过优化实现 3D 生成，最终实现了高质量、可控的 3D 内容的生成。

可控的文本到三维生成：面对齐的高斯扩散

Controllable Text-to-3D Generation via Surface-Aligned Gaussian  Splatting

Recent advancements in text-to-3D generation have significantly contributed
to the automation and democratization of 3D content creation. Building upon
these developments, we aim to address the limitations of current methods in
generating 3D models with creative geometry and styles. We introduce multi-view
ControlNet, a novel depth-aware multi-view diffusion model trained on generated
datasets from a carefully curated 100K text corpus. Our multi-view ControlNet
is then integrated into our two-stage pipeline, ControlDreamer, enabling
text-guided generation of stylized 3D models. Additionally, we present a
comprehensive benchmark for 3D style editing, encompassing a broad range of
subjects, including objects, animals, and characters, to further facilitate
diverse 3D generation. Our comparative analysis reveals that this new pipeline
outperforms existing text-to-3D methods as evidenced by qualitative comparisons
and CLIP score metrics.

通过引入多视角 ControlNet 和强化学习模型 ControlDreamer，本研究在文本指导下实现了创意几何和风格的 3D 建模，同时提出了一个全面的 3D 风格编辑基准，证明该方法在质量和度量上优于现有的文本生成 3D 方法。