This paper aims to generate materials for 3D meshes from text descriptions.
Unlike existing methods that synthesize texture maps, we propose to generate
segment-wise procedural material graphs as the appearance representation, which
supports high-quality rendering and provides substantial flexibility in
editing. Instead of relying on extensive paired data, i.e., 3D meshes with
material graphs and corresponding text descriptions, to train a material graph
generative model, we propose to leverage the pre-trained 2D diffusion model as
a bridge to connect the text and material graphs. Specifically, our approach
decomposes a shape into a set of segments and designs a segment-controlled
diffusion model to synthesize 2D images that are aligned with mesh parts. Based
on generated images, we initialize parameters of material graphs and fine-tune
them through the differentiable rendering module to produce materials in
accordance with the textual description. Extensive experiments demonstrate the
superior performance of our framework in photorealism, resolution, and
editability over existing methods. Project page:
this https URL

本文旨在通过文本描述生成 3D 网格材料。与现有方法合成纹理图不同，我们提出生成分段逐步建模材料图作为外观表示，支持高质量渲染并提供灵活的编辑。我们的方法通过预训练的 2D 扩散模型作为文本和材料图之间的桥梁，而不是依赖于大量的成对数据训练材料图生成模型。通过在生成的图像上初始化材料图参数，并通过可微分渲染模块进行微调，以符合文本描述。大量实验证明我们的框架在逼真度、分辨率和可编辑性方面的性能优于现有方法。

MaPa：基于文本的用于 3D 形状的逼真材质绘制

MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

Most indoor 3D scene reconstruction methods focus on recovering 3D geometry
and scene layout. In this work, we go beyond this to propose PhotoScene, a
framework that takes input image(s) of a scene along with approximately aligned
CAD geometry (either reconstructed automatically or manually specified) and
builds a photorealistic digital twin with high-quality materials and similar
lighting. We model scene materials using procedural material graphs; such
graphs represent photorealistic and resolution-independent materials. We
optimize the parameters of these graphs and their texture scale and rotation,
as well as the scene lighting to best match the input image via a
differentiable rendering layer. We evaluate our technique on objects and layout
reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate
that our method reconstructs high-quality, fully relightable 3D scenes that can
be re-rendered under arbitrary viewpoints, zooms and lighting.

本文提出了一种名为 PhotoScene 的框架，可以基于输入图像及与之大致对齐的 CAD 几何形态数据（自动重建或手动指定），构建具有高质量材质和相似照明的逼真数字孪生体。该框架采用过程材料图模型建模场景材料，可以优化图参数和纹理比例、旋转，以及场景光照度以最佳匹配输入图像。文中还对该技术进行了评估，并展示了该方法对于来自 ScanNet、SUN RGB-D 和证券照片的对象和布局重建可以重渲染出高质量、可完全照明的 3D 场景。