In this paper, we propose a method to extract physically-based rendering
(PBR) materials from a single real-world image. We do so in two steps: first,
we map regions of the image to material concepts using a diffusion model, which
allows the sampling of texture images resembling each material in the scene.
Second, we benefit from a separate network to decompose the generated textures
into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be
used in rendering applications. Our approach builds on existing synthetic
material libraries with SVBRDF ground truth, but also exploits a
diffusion-generated RGB texture dataset to allow generalization to new samples
using unsupervised domain adaptation (UDA). Our contributions are thoroughly
evaluated on synthetic and real-world datasets. We further demonstrate the
applicability of our method for editing 3D scenes with materials estimated from
real photographs. The code and models will be made open-source. Project page:
this https URL

本研究提出了一种从单个真实世界图像中提取基于物理的渲染 (PBR) 材料的方法，通过扩散模型将图像的区域映射到材料概念，然后利用独立的网络将生成的纹理分解成空间变化的 BRDFs，并通过无监督领域适应实现对新样本的泛化应用。该方法在合成和真实世界数据集上进行了充分评估，并展示了从实际照片估计的材料编辑 3D 场景的适用性。