Recent remarkable advances in large-scale text-to-image diffusion models have
inspired a significant breakthrough in text-to-3D generation, pursuing 3D
content creation solely from a given text prompt. However, existing text-to-3D
techniques lack a crucial ability in the creative process: interactively
control and shape the synthetic 3D contents according to users' desired
specifications (e.g., sketch). To alleviate this issue, we present the first
attempt for text-to-3D generation conditioning on the additional hand-drawn
sketch, namely Control3D, which enhances controllability for users. In
particular, a 2D conditioned diffusion model (ControlNet) is remoulded to guide
the learning of 3D scene parameterized as NeRF, encouraging each view of 3D
scene aligned with the given text prompt and hand-drawn sketch. Moreover, we
exploit a pre-trained differentiable photo-to-sketch model to directly estimate
the sketch of the rendered image over synthetic 3D scene. Such estimated sketch
along with each sampled view is further enforced to be geometrically consistent
with the given sketch, pursuing better controllable text-to-3D generation.
Through extensive experiments, we demonstrate that our proposal can generate
accurate and faithful 3D scenes that align closely with the input text prompts
and sketches.

大规模文本到图像扩散模型的最新进展在文本到三维生成领域取得了重大突破，仅通过给定的文本提示从零开始创作三维内容。然而，现有的文本到三维技术在创作过程中缺乏一项关键能力：根据用户的需求规范（如草图）对合成的三维内容进行交互式控制和塑造。为了解决这个问题，我们首次尝试在条件上添加手绘草图的文本到三维生成，即 Control3D，以增强用户的可控性。具体而言，我们通过改进的 2D 条件扩散模型（ControlNet）来引导作为 NeRF 参数化的三维场景的学习，以使每个三维场景视角与给定的文本提示和手绘草图对齐。此外，我们利用预训练的可微分照片到草图模型直接估计合成三维场景上渲染图像的草图。此类估计的草图以及每个采样视角进一步被强制与给定的草图在几何上保持一致，从而实现了更好的可控文本到三维生成。通过广泛的实验证明，我们的提议可以生成与输入的文本提示和草图紧密对齐的准确忠实的三维场景。