2D diffusion model, which often contains unwanted baked-in shading effects
and results in unrealistic rendering effects in the downstream applications.
Generating Physically Based Rendering (PBR) materials instead of just RGB
textures would be a promising solution. However, directly distilling the PBR
material parameters from 2D diffusion models still suffers from incorrect
material decomposition, such as baked-in shading effects in albedo. We
introduce DreamMat, an innovative approach to resolve the aforementioned
problem, to generate high-quality PBR materials from text descriptions. We find
out that the main reason for the incorrect material distillation is that
large-scale 2D diffusion models are only trained to generate final shading
colors, resulting in insufficient constraints on material decomposition during
distillation. To tackle this problem, we first finetune a new light-aware 2D
diffusion model to condition on a given lighting environment and generate the
shading results on this specific lighting condition. Then, by applying the same
environment lights in the material distillation, DreamMat can generate
high-quality PBR materials that are not only consistent with the given geometry
but also free from any baked-in shading effects in albedo. Extensive
experiments demonstrate that the materials produced through our methods exhibit
greater visual appeal to users and achieve significantly superior rendering
quality compared to baseline methods, which are preferable for downstream tasks
such as game and film production.

通过 DreamMat 方法从文本描述生成高质量 PBR 材料，解决了从 2D 扩散模型中提取 PBR 材料参数时的错误材料解析问题，并取得了显著的渲染质量优势。