Recent advances in text-to-3D generation have been remarkable, with methods
such as DreamFusion leveraging large-scale text-to-image diffusion-based models
to supervise 3D generation. These methods, including the variational score
distillation proposed by ProlificDreamer, enable the synthesis of detailed and
photorealistic textured meshes. However, the appearance of 3D objects generated
by these methods is often random and uncontrollable, posing a challenge in
achieving appearance-controllable 3D objects. To address this challenge, we
introduce IPDreamer, a novel approach that incorporates image prompts to
provide specific and comprehensive appearance information for 3D object
generation. Our results demonstrate that IPDreamer effectively generates
high-quality 3D objects that are consistent with both the provided text and
image prompts, demonstrating its promising capability in
appearance-controllable 3D object generation.

最新的文字到 3D 生成技术取得了显著进展，现有的方法利用大规模的文字到图像扩散模型，如 DreamFusion，来监督 3D 生成。然而，这些方法生成的 3D 对象的外观通常是随机和不可控的，这在实现可控外观的 3D 对象方面是一个挑战。为了解决这个问题，我们引入了 IPDreamer，一种新颖的方法，它结合了图像提示，为 3D 对象生成提供特定和全面的外观信息。我们的结果表明，IPDreamer 有效地生成了与所提供的文本和图像提示一致的高质量 3D 对象，展示了它在可控外观的 3D 对象生成方面的潜能。

IPDreamer：基于图像提示的外观可控 3D 物体生成

IPDreamer: Appearance-Controllable 3D Object Generation with Image  Prompts

Although text-to-image diffusion models have made significant strides in
generating images from text, they are sometimes more inclined to generate
images like the data on which the model was trained rather than the provided
text. This limitation has hindered their usage in both 2D and 3D applications.
To address this problem, we explored the use of negative prompts but found that
the current implementation fails to produce desired results, particularly when
there is an overlap between the main and negative prompts. To overcome this
issue, we propose Perp-Neg, a new algorithm that leverages the geometrical
properties of the score space to address the shortcomings of the current
negative prompts algorithm. Perp-Neg does not require any training or
fine-tuning of the model. Moreover, we experimentally demonstrate that Perp-Neg
provides greater flexibility in generating images by enabling users to edit out
unwanted concepts from the initially generated images in 2D cases. Furthermore,
to extend the application of Perp-Neg to 3D, we conducted a thorough
exploration of how Perp-Neg can be used in 2D to condition the diffusion model
to generate desired views, rather than being biased toward the canonical views.
Finally, we applied our 2D intuition to integrate Perp-Neg with the
state-of-the-art text-to-3D (DreamFusion) method, effectively addressing its
Janus (multi-head) problem. Our project page is available at
this https URL

本文介绍了一种名为 Perp-Neg 的算法，它可以成功地使用负提示来制定文本到图像扩散模型，为 2D 和 3D 应用提供了更大的灵活性，并成功地解决了文本到 3D 方法的 Janus 问题。