Prior material creation methods had limitations in producing diverse results
mainly because reconstruction-based methods relied on real-world measurements
and generation-based methods were trained on relatively small material
datasets. To address these challenges, we propose DreamPBR, a novel
diffusion-based generative framework designed to create spatially-varying
appearance properties guided by text and multi-modal controls, providing high
controllability and diversity in material generation. Key to achieving diverse
and high-quality PBR material generation lies in integrating the capabilities
of recent large-scale vision-language models trained on billions of text-image
pairs, along with material priors derived from hundreds of PBR material
samples. We utilize a novel material Latent Diffusion Model (LDM) to establish
the mapping between albedo maps and the corresponding latent space. The latent
representation is then decoded into full SVBRDF parameter maps using a
rendering-aware PBR decoder. Our method supports tileable generation through
convolution with circular padding. Furthermore, we introduce a multi-modal
guidance module, which includes pixel-aligned guidance, style image guidance,
and 3D shape guidance, to enhance the control capabilities of the material LDM.
We demonstrate the effectiveness of DreamPBR in material creation, showcasing
its versatility and user-friendliness on a wide range of controllable
generation and editing applications.

通过结合大规模的视觉语言模型和已存在的先验知识，我们提出了 DreamPBR，一种新颖的基于扩散的生成框架，用于创建由文本和多模态控制引导的具有空间变化外观属性的材料，从而提供高度可控性和多样性。通过我们的方法，在材料 creation 和编辑应用方面展示了 DreamPBR 的多功能性和用户友好性。